[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:35 dist-test-slave-c60x krb5kdc[6277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:35 dist-test-slave-c60x krb5kdc[6277](info): set up 2 sockets
Sep 27 10:49:35 dist-test-slave-c60x krb5kdc[6277](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 10:49:36.787855 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:49:36 dist-test-slave-c60x krb5kdc[6277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:49:36.919252 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37447
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:37.028098 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:37.028385 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:37.028463 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:37.032444 6290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:37.032522 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:37.032560 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:37.032618 6290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:37.032665 6290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:37.037784 6290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:37.038806 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:37.039873 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:37.048225 6296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:37.048174 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:37.048188 6295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:37.048507 6290 server_base.cc:1034] running on GCE node
I20260927 10:49:37.049126 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:37.050261 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:37.051440 6290 hybrid_clock.cc:648] HybridClock initialized: now 1790506177051410 us; error 42 us; skew 500 ppm
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:37.054350 6290 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:49:37.055420 6290 webserver.cc:460] Webserver started at http://127.6.31.190:37913/ using document root <none> and password file <none>
I20260927 10:49:37.056015 6290 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:37.056107 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:37.056339 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:37.058391 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "e44f49ffb3684a05919969f65dc9e5ac"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "ef00101ffaebce5e30ecdf4c0bdf6f18"
server_key_iv: "883d6cae4bed7a3a582aa8c3c08fa94e"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.058974 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "e44f49ffb3684a05919969f65dc9e5ac"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "ef00101ffaebce5e30ecdf4c0bdf6f18"
server_key_iv: "883d6cae4bed7a3a582aa8c3c08fa94e"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.062960 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:49:37.065548 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.066811 6290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:37.067061 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "e44f49ffb3684a05919969f65dc9e5ac"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "ef00101ffaebce5e30ecdf4c0bdf6f18"
server_key_iv: "883d6cae4bed7a3a582aa8c3c08fa94e"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.067257 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:37.095738 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:37.098950 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:37.099192 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:37.106413 6290 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:37447
I20260927 10:49:37.106434 6357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:37447 every 8 connection(s)
I20260927 10:49:37.107414 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:37.110217 6358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:37.116040 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 6290
I20260927 10:49:37.116230 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:49:37.116415 6358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac: Bootstrap starting.
I20260927 10:49:37.117548 6270 external_mini_cluster.cc:1428] Setting key c52a3a35d0c1e4741ac6f56621f54532
I20260927 10:49:37.119413 6358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:37.120188 6358 log.cc:824] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:37.122159 6358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac: No bootstrap required, opened a new log
I20260927 10:49:37.126266 6358 raft_consensus.cc:348] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f49ffb3684a05919969f65dc9e5ac" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37447 } }
I20260927 10:49:37.126539 6358 raft_consensus.cc:374] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:37.126605 6358 raft_consensus.cc:729] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44f49ffb3684a05919969f65dc9e5ac, State: Initialized, Role: FOLLOWER
I20260927 10:49:37.127074 6358 consensus_queue.cc:260] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f49ffb3684a05919969f65dc9e5ac" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37447 } }
I20260927 10:49:37.127264 6358 raft_consensus.cc:388] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:37.127351 6358 raft_consensus.cc:482] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:37.127476 6358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:37.128388 6358 raft_consensus.cc:504] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f49ffb3684a05919969f65dc9e5ac" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37447 } }
I20260927 10:49:37.128732 6358 leader_election.cc:302] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e44f49ffb3684a05919969f65dc9e5ac; no voters:
I20260927 10:49:37.129055 6358 leader_election.cc:288] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:37.129123 6363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:37.129313 6363 raft_consensus.cc:686] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [term 1 LEADER]: Becoming Leader. State: Replica: e44f49ffb3684a05919969f65dc9e5ac, State: Running, Role: LEADER
I20260927 10:49:37.129580 6363 consensus_queue.cc:237] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f49ffb3684a05919969f65dc9e5ac" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37447 } }
I20260927 10:49:37.130306 6358 sys_catalog.cc:564] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:37.131058 6365 sys_catalog.cc:455] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e44f49ffb3684a05919969f65dc9e5ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f49ffb3684a05919969f65dc9e5ac" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37447 } } }
I20260927 10:49:37.131384 6364 sys_catalog.cc:455] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e44f49ffb3684a05919969f65dc9e5ac. Latest consensus state: current_term: 1 leader_uuid: "e44f49ffb3684a05919969f65dc9e5ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f49ffb3684a05919969f65dc9e5ac" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37447 } } }
I20260927 10:49:37.131507 6365 sys_catalog.cc:458] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:37.131553 6364 sys_catalog.cc:458] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac [sys.catalog]: This master's current role is: LEADER
W20260927 10:49:37.135135 6378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:37.135218 6378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:37.136533 6373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:37.137428 6373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:37.143504 6373 catalog_manager.cc:1347] Generated new cluster ID: 37b2ba7999214aa79be9a8c4c27a9b8e
I20260927 10:49:37.143587 6373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:37.144186 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:37.120795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.6.31.190:37447)
0927 10:49:37.121249 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:37.121268 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:37.122480 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 10:49:37.125805 (+ 3325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:37.125826 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:37.126205 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:37.126557 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:37.129195 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.130471 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.132305 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.132987 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.133151 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:37.137854 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:37.137859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:37.137870 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:37.137895 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:37.139889 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.140538 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.140544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.140548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.140702 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.141156 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.141160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.141162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.141680 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:37.141876 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:37.142534 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 10:49:37.142836 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260927 10:49:37.158447 6373 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:37.159427 6373 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:37.164119 6373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e44f49ffb3684a05919969f65dc9e5ac: Generated new TSK 0
I20260927 10:49:37.164777 6373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:49:37.199505 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37447
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:37.328477 6384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:37.328809 6384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:37.328893 6384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:37.333381 6384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:37.333474 6384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:37.333568 6384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:49:37.338949 6384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:37.340113 6384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:37.341161 6384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:37.350582 6389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:37.350603 6390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:37.350940 6384 server_base.cc:1034] running on GCE node
W20260927 10:49:37.351101 6392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:37.351473 6384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:37.352180 6384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:37.353348 6384 hybrid_clock.cc:648] HybridClock initialized: now 1790506177353325 us; error 33 us; skew 500 ppm
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:37.356487 6384 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:49:37.357391 6384 webserver.cc:460] Webserver started at http://127.6.31.129:42567/ using document root <none> and password file <none>
I20260927 10:49:37.357959 6384 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:37.358019 6384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:37.358278 6384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:37.360415 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "97d53ae8bb4a421499a68d4363277b3f"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "66426f9dec6b99a0827c4c4f82db8c90"
server_key_iv: "22674d5148ba7d16b42524913be9f9de"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.360981 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "97d53ae8bb4a421499a68d4363277b3f"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "66426f9dec6b99a0827c4c4f82db8c90"
server_key_iv: "22674d5148ba7d16b42524913be9f9de"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.364853 6384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:49:37.367401 6399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.368651 6384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:49:37.368777 6384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "97d53ae8bb4a421499a68d4363277b3f"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "66426f9dec6b99a0827c4c4f82db8c90"
server_key_iv: "22674d5148ba7d16b42524913be9f9de"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.368891 6384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:37.383832 6384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:37.386863 6384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:37.387111 6384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:37.387740 6384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:37.388557 6384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:37.388607 6384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.388648 6384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:37.388703 6384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.398353 6384 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:41057
I20260927 10:49:37.398375 6512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:41057 every 8 connection(s)
I20260927 10:49:37.399293 6384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:49:37.406838 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 6384
I20260927 10:49:37.406930 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:49:37.407264 6270 external_mini_cluster.cc:1428] Setting key 4c6845b7c641b38aa8566665a8f1a6ba
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:37.414894 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:37.401227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:47115 (local address 127.6.31.190:37447)
0927 10:49:37.401381 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:37.401385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:37.402215 (+ 830us) server_negotiation.cc:407] Connection header received
0927 10:49:37.403702 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:37.403706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:37.403773 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:37.403882 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:37.405065 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.405606 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.406755 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.407070 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.407220 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:37.410471 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:37.410474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:37.410477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:37.410510 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:37.412443 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.413216 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.413219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.413221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.413311 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.413733 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.413736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.413737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.413908 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:37.413985 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:37.414664 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 10:49:37.414771 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:49:37.416257 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:37.401609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37447 (local address 127.6.31.129:47115)
0927 10:49:37.402060 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:37.402102 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:37.403473 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:37.404056 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:37.404063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:37.404437 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:37.404915 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:37.404928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.405758 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.405761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:37.406625 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:37.406635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.407261 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.407266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:37.407485 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:37.408519 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:37.408536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:37.410278 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:37.412585 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:37.412593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:37.412614 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:37.413000 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:37.413417 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:37.413423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:37.413429 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:37.413641 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:37.414016 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:37.414024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:37.414382 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 10:49:37.414633 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 10:49:37.415026 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":125,"threads_started":1}
I20260927 10:49:37.417579 6513 heartbeater.cc:347] Connected to a master server at 127.6.31.190:37447
I20260927 10:49:37.417831 6513 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:49:37.418375 6513 heartbeater.cc:511] Master 127.6.31.190:37447 requested a full tablet report, sending...
I20260927 10:49:37.419955 6322 ts_manager.cc:194] Registered new tserver with Master: 97d53ae8bb4a421499a68d4363277b3f (127.6.31.129:41057)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:49:37.421375 6322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:47115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:49:37.439052 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37447
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:37.550063 6518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:37.550379 6518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:37.550460 6518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:37.554669 6518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:37.554744 6518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:37.554831 6518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:49:37.560181 6518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:37.561260 6518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:37.562276 6518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:37.571949 6524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:37.572049 6523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:37.571985 6526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:37.572252 6518 server_base.cc:1034] running on GCE node
I20260927 10:49:37.572698 6518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:37.573374 6518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:37.574568 6518 hybrid_clock.cc:648] HybridClock initialized: now 1790506177574554 us; error 27 us; skew 500 ppm
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:37.577412 6518 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:49:37.578344 6518 webserver.cc:460] Webserver started at http://127.6.31.130:42891/ using document root <none> and password file <none>
I20260927 10:49:37.578905 6518 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:37.578994 6518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:37.579252 6518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:37.581218 6518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "e7d30b8efdbd48c497251c87761659be"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "b16afbb1d010ccf1e5d826521b14614b"
server_key_iv: "1bbd448473d75fe9a753ecce40ee9726"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.581770 6518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "e7d30b8efdbd48c497251c87761659be"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "b16afbb1d010ccf1e5d826521b14614b"
server_key_iv: "1bbd448473d75fe9a753ecce40ee9726"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.585417 6518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:49:37.587831 6533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.589016 6518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:37.589147 6518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "e7d30b8efdbd48c497251c87761659be"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "b16afbb1d010ccf1e5d826521b14614b"
server_key_iv: "1bbd448473d75fe9a753ecce40ee9726"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.589253 6518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:37.601706 6518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:37.604691 6518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:37.604882 6518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:37.605438 6518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:37.606217 6518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:37.606287 6518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.606350 6518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:37.606402 6518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.616196 6518 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:43013
I20260927 10:49:37.616225 6646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:43013 every 8 connection(s)
I20260927 10:49:37.617026 6518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:49:37.625427 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 6518
I20260927 10:49:37.625571 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:49:37.625962 6270 external_mini_cluster.cc:1428] Setting key 9b40d19bfa3ae6dbcff20c78313e4b61
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:37.631042 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:37.618797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:55511 (local address 127.6.31.190:37447)
0927 10:49:37.618967 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:37.618972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:37.619785 (+ 813us) server_negotiation.cc:407] Connection header received
0927 10:49:37.621084 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:37.621088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:37.621146 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:37.621254 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:37.622391 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.622945 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.624065 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.624277 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.624357 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:37.626992 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:37.626996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:37.626999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:37.627033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:37.628575 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.629234 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.629239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.629244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.629354 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.629693 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.629698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.629702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.629947 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:37.630067 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:37.630735 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 10:49:37.630861 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 10:49:37.631978 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:37.619204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37447 (local address 127.6.31.130:55511)
0927 10:49:37.619624 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:37.619665 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:37.620845 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:37.621389 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:37.621395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:37.621794 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:37.622254 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:37.622266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.623048 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.623050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:37.623961 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:37.623970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.624404 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.624406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:37.624561 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:37.625108 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:37.625124 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:37.626822 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:37.628691 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:37.628701 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:37.628720 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:37.629091 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:37.629460 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:37.629462 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:37.629463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:37.629592 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:37.630109 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:37.630124 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:37.630412 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 10:49:37.630682 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 10:49:37.631021 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260927 10:49:37.633087 6647 heartbeater.cc:347] Connected to a master server at 127.6.31.190:37447
I20260927 10:49:37.633344 6647 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:37.633847 6647 heartbeater.cc:511] Master 127.6.31.190:37447 requested a full tablet report, sending...
I20260927 10:49:37.634881 6322 ts_manager.cc:194] Registered new tserver with Master: e7d30b8efdbd48c497251c87761659be (127.6.31.130:43013)
I20260927 10:49:37.635502 6322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:55511
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:49:37.654825 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37447
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:37.764343 6652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:37.764583 6652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:37.764627 6652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:37.768567 6652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:37.768625 6652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:37.768689 6652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:49:37.774134 6652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:37.775163 6652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:37.776130 6652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:37.785415 6660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:37.785404 6657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:37.785401 6658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:37.785953 6652 server_base.cc:1034] running on GCE node
I20260927 10:49:37.786300 6652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:37.786918 6652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:37.788098 6652 hybrid_clock.cc:648] HybridClock initialized: now 1790506177788068 us; error 48 us; skew 500 ppm
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:37.791400 6652 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:49:37.792357 6652 webserver.cc:460] Webserver started at http://127.6.31.131:44423/ using document root <none> and password file <none>
I20260927 10:49:37.792968 6652 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:37.793030 6652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:37.793253 6652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:37.795465 6652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "fe32e69657424413bb9c2613b9c828ac"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "c27fd355079f34901c9a8074a5ef9146"
server_key_iv: "efd95db797a285bd7f8b3064639702a1"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.795996 6652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "fe32e69657424413bb9c2613b9c828ac"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "c27fd355079f34901c9a8074a5ef9146"
server_key_iv: "efd95db797a285bd7f8b3064639702a1"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.799772 6652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:49:37.802227 6667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.803668 6652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:49:37.803802 6652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "fe32e69657424413bb9c2613b9c828ac"
format_stamp: "Formatted at 2026-09-27 10:49:37 on dist-test-slave-c60x"
server_key: "c27fd355079f34901c9a8074a5ef9146"
server_key_iv: "efd95db797a285bd7f8b3064639702a1"
server_key_version: "encryptionkey@0"
I20260927 10:49:37.803915 6652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:37.838317 6652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:37.841415 6652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:37.841641 6652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:37.842248 6652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:37.843108 6652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:37.843161 6652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.843226 6652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:37.843261 6652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:37.852979 6652 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:46491
I20260927 10:49:37.853001 6780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:46491 every 8 connection(s)
I20260927 10:49:37.853855 6652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:49:37.861182 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 6652
I20260927 10:49:37.861321 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:49:37.861586 6270 external_mini_cluster.cc:1428] Setting key e855f97f2db51eba36b0aa5e8fc5bb6c
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:37.868080 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:37.855677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:40711 (local address 127.6.31.190:37447)
0927 10:49:37.855828 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:37.855832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:37.857081 (+ 1249us) server_negotiation.cc:407] Connection header received
0927 10:49:37.857992 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:37.857996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:37.858055 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:37.858133 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:37.859309 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.859843 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.860975 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.861202 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.861318 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:37.864269 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:37.864273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:37.864275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:37.864304 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:37.865808 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.866352 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.866355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.866356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.866437 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.866775 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.866778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.866779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.867175 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:37.867261 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:37.867842 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 10:49:37.867955 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:49:37.868989 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:37.856064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37447 (local address 127.6.31.131:40711)
0927 10:49:37.856938 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:37.856978 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:37.857794 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:37.858293 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:37.858299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:37.858641 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:37.859165 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:37.859181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.859984 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.859988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:37.860837 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:37.860845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.861351 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.861355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:37.861526 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:37.862216 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:37.862240 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:37.864099 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:37.865924 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:37.865929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:37.865941 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:37.866209 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:37.866530 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:37.866532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:37.866534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:37.866673 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:37.867281 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:37.867287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:37.867525 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 10:49:37.867761 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 10:49:37.868105 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":564,"threads_started":1}
I20260927 10:49:37.870293 6781 heartbeater.cc:347] Connected to a master server at 127.6.31.190:37447
I20260927 10:49:37.870594 6781 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:37.871160 6781 heartbeater.cc:511] Master 127.6.31.190:37447 requested a full tablet report, sending...
I20260927 10:49:37.872346 6322 ts_manager.cc:194] Registered new tserver with Master: fe32e69657424413bb9c2613b9c828ac (127.6.31.131:46491)
I20260927 10:49:37.873099 6322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:40711
I20260927 10:49:37.876848 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:49:37.903502 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:37.893873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.6.31.129:41057)
0927 10:49:37.894232 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:37.894247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:37.894276 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:49:37.894343 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:37.894350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:37.894551 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:37.894710 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:37.895477 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.896273 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.897109 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.897485 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.897603 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:37.900175 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:37.900199 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:37.900212 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:37.900236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:37.901738 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.902148 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.902152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.902153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.902239 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.902527 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.902529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.902530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.902771 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:37.902930 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:37.903220 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 10:49:37.903348 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
W20260927 10:49:37.904592 6427 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47112
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:49:37 dist-test-slave-c60x krb5kdc[6277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:49:37.929838 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:37.921706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.6.31.129:41057)
0927 10:49:37.921902 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:37.921908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:37.921934 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:49:37.922068 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:37.922071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:37.922112 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:37.922183 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:37.922835 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.923399 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.924157 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:37.924369 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:37.924442 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:37.926690 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:37.926693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:37.926695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:37.926717 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:37.928175 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.928522 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.928525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.928526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.928652 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:37.928886 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:37.928889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:37.928890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:37.929071 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:37.929170 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:37.929401 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 10:49:37.929493 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:49:37.930960 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 6384
I20260927 10:49:37.936655 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 6518
I20260927 10:49:37.942363 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 6652
I20260927 10:49:37.947558 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 6290
2026-09-27T10:49:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2226 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:38 dist-test-slave-c60x krb5kdc[6795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:38 dist-test-slave-c60x krb5kdc[6795](info): set up 2 sockets
Sep 27 10:49:38 dist-test-slave-c60x krb5kdc[6795](info): commencing operation
krb5kdc: starting...
W20260927 10:49:38.990859 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:49:39 dist-test-slave-c60x krb5kdc[6795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:49:39.114248 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33909
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:39.225446 6808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:39.225713 6808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:39.225752 6808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:39.229806 6808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:39.229859 6808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:39.229879 6808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:39.229897 6808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:39.229954 6808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:39.235260 6808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:39.236305 6808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:39.237326 6808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:39.245514 6816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:39.245522 6813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:39.245612 6814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:39.245697 6808 server_base.cc:1034] running on GCE node
I20260927 10:49:39.246320 6808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:39.247483 6808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:39.248682 6808 hybrid_clock.cc:648] HybridClock initialized: now 1790506179248655 us; error 42 us; skew 500 ppm
Sep 27 10:49:39 dist-test-slave-c60x krb5kdc[6795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:39.251571 6808 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:49:39.252342 6808 webserver.cc:460] Webserver started at http://127.6.31.190:42231/ using document root <none> and password file <none>
I20260927 10:49:39.252851 6808 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:39.252903 6808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:39.253058 6808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:39.255028 6808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "3e21201ab6ac468880aacffcf89c5c60"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "8d43b9e4358e282e399c9c72b2d46f27"
server_key_iv: "7912ca35afa864ff26cba000b45f7d8d"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.255625 6808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "3e21201ab6ac468880aacffcf89c5c60"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "8d43b9e4358e282e399c9c72b2d46f27"
server_key_iv: "7912ca35afa864ff26cba000b45f7d8d"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.259408 6808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:49:39.261729 6823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:39.262956 6808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:39.263087 6808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "3e21201ab6ac468880aacffcf89c5c60"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "8d43b9e4358e282e399c9c72b2d46f27"
server_key_iv: "7912ca35afa864ff26cba000b45f7d8d"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.263283 6808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:39.282802 6808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:39.286340 6808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:39.286541 6808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:39.293418 6808 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:33909
I20260927 10:49:39.293485 6875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:33909 every 8 connection(s)
I20260927 10:49:39.294375 6808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:39.297152 6876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:39.300537 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 6808
I20260927 10:49:39.300686 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:49:39.300949 6270 external_mini_cluster.cc:1428] Setting key a76993ce1fa4020413b6b65898fe450d
I20260927 10:49:39.302819 6876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60: Bootstrap starting.
I20260927 10:49:39.305269 6876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:39.305977 6876 log.cc:824] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:39.308137 6876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60: No bootstrap required, opened a new log
Sep 27 10:49:39 dist-test-slave-c60x krb5kdc[6795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:39.311443 6876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e21201ab6ac468880aacffcf89c5c60" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33909 } }
I20260927 10:49:39.311688 6876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:39.311770 6876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e21201ab6ac468880aacffcf89c5c60, State: Initialized, Role: FOLLOWER
I20260927 10:49:39.312186 6876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e21201ab6ac468880aacffcf89c5c60" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33909 } }
I20260927 10:49:39.312340 6876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:39.312423 6876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:39.312541 6876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:39.313390 6876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e21201ab6ac468880aacffcf89c5c60" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33909 } }
I20260927 10:49:39.313723 6876 leader_election.cc:302] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e21201ab6ac468880aacffcf89c5c60; no voters:
I20260927 10:49:39.314045 6876 leader_election.cc:288] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:39.314116 6881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:39.314301 6881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [term 1 LEADER]: Becoming Leader. State: Replica: 3e21201ab6ac468880aacffcf89c5c60, State: Running, Role: LEADER
I20260927 10:49:39.314638 6881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e21201ab6ac468880aacffcf89c5c60" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33909 } }
I20260927 10:49:39.315392 6876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:39.316207 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:39.302339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.6.31.190:33909)
0927 10:49:39.302730 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:39.302745 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:39.302779 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:49:39.303475 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:39.303492 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:39.303797 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:39.304051 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:39.304899 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.305929 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.306992 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.307512 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.307629 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:39.310344 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:39.310349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:39.310362 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:39.310388 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:39.312778 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:39.313239 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:39.313245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:39.313251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:39.313365 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:39.313644 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:39.313649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:39.313650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:39.314137 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:39.314873 (+ 736us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:39.315044 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 10:49:39.315345 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":127,"threads_started":1}
I20260927 10:49:39.316457 6883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e21201ab6ac468880aacffcf89c5c60. Latest consensus state: current_term: 1 leader_uuid: "3e21201ab6ac468880aacffcf89c5c60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e21201ab6ac468880aacffcf89c5c60" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33909 } } }
I20260927 10:49:39.316569 6883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:39.316941 6882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e21201ab6ac468880aacffcf89c5c60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e21201ab6ac468880aacffcf89c5c60" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33909 } } }
I20260927 10:49:39.317884 6882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60 [sys.catalog]: This master's current role is: LEADER
W20260927 10:49:39.320179 6897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:39.320250 6897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:39.320326 6890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:39.320995 6890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:39.326924 6890 catalog_manager.cc:1347] Generated new cluster ID: ba6b8bd30fa442c2bc522404eae54ff4
I20260927 10:49:39.327020 6890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:39.343153 6890 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:39.344159 6890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:39.351895 6890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e21201ab6ac468880aacffcf89c5c60: Generated new TSK 0
I20260927 10:49:39.352617 6890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:49:39.388023 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33909
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:39.505280 6902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:39.505535 6902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:39.505577 6902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:39.509528 6902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:39.509588 6902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:39.509646 6902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:49:39.514846 6902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33909
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:39.516085 6902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:39.517205 6902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:39.526592 6908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:39.526716 6910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:39.526796 6907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:39.526980 6902 server_base.cc:1034] running on GCE node
I20260927 10:49:39.527402 6902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:39.528023 6902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:39.529189 6902 hybrid_clock.cc:648] HybridClock initialized: now 1790506179529177 us; error 24 us; skew 500 ppm
Sep 27 10:49:39 dist-test-slave-c60x krb5kdc[6795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:39.531953 6902 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:49:39.532805 6902 webserver.cc:460] Webserver started at http://127.6.31.129:44137/ using document root <none> and password file <none>
I20260927 10:49:39.533347 6902 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:39.533404 6902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:39.533607 6902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:39.535648 6902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "39b83d0b36934ff8973130bd1a5e8347"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "2f24ac737ada64c8e70996017d3337d7"
server_key_iv: "f7606b4bc57ea8ea3a72a4086e8295ca"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.536192 6902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "39b83d0b36934ff8973130bd1a5e8347"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "2f24ac737ada64c8e70996017d3337d7"
server_key_iv: "f7606b4bc57ea8ea3a72a4086e8295ca"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.539980 6902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:49:39.542366 6917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:39.543653 6902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:39.543826 6902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "39b83d0b36934ff8973130bd1a5e8347"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "2f24ac737ada64c8e70996017d3337d7"
server_key_iv: "f7606b4bc57ea8ea3a72a4086e8295ca"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.543931 6902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:39.560920 6902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:39.564009 6902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:39.564205 6902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:39.564774 6902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:39.565559 6902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:39.565627 6902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:39.565718 6902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:39.565765 6902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:39.574466 6902 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:37979
I20260927 10:49:39.574483 7030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:37979 every 8 connection(s)
I20260927 10:49:39.575424 6902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:49:39.585079 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 6902
I20260927 10:49:39.585171 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:49:39.585438 6270 external_mini_cluster.cc:1428] Setting key 050e865950f04ee2cd23bc2b57191dfd
Sep 27 10:49:39 dist-test-slave-c60x krb5kdc[6795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:39.589686 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:39.577271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:57129 (local address 127.6.31.190:33909)
0927 10:49:39.577417 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:39.577425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:39.578276 (+ 851us) server_negotiation.cc:407] Connection header received
0927 10:49:39.579875 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:39.579877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:39.579923 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:39.580006 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:39.581211 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.581747 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.582885 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.583389 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.583476 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:39.586101 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:39.586105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:39.586107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:39.586153 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:39.587625 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:39.588188 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:39.588192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:39.588196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:39.588274 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:39.588655 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:39.588657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:39.588659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:39.588825 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:39.588933 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:39.589397 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 10:49:39.589521 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:49:39.590469 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:39.577578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33909 (local address 127.6.31.129:57129)
0927 10:49:39.578096 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:39.578151 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:39.579666 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:39.580166 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:39.580172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:39.580526 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:39.581052 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:39.581065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.581874 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.581877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:39.582771 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:39.582780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.583523 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.583526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:39.583664 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:39.584304 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:39.584320 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:39.585913 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:39.587761 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:39.587767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:39.587787 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:39.588055 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:39.588367 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:39.588371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:39.588373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:39.588508 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:39.588931 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:39.588938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:39.589174 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 10:49:39.589364 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 10:49:39.589679 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":100,"threads_started":1}
I20260927 10:49:39.591665 7031 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33909
I20260927 10:49:39.591926 7031 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:39.592519 7031 heartbeater.cc:511] Master 127.6.31.190:33909 requested a full tablet report, sending...
I20260927 10:49:39.594301 6837 ts_manager.cc:194] Registered new tserver with Master: 39b83d0b36934ff8973130bd1a5e8347 (127.6.31.129:37979)
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:49:39.595705 6837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:57129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:49:39.613544 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33909
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:39.722136 7036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:39.722388 7036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:39.722431 7036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:39.726490 7036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:39.726547 7036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:39.726641 7036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:49:39.731817 7036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33909
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:39.732800 7036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:39.733824 7036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:49:39.743287 7036 server_base.cc:1034] running on GCE node
W20260927 10:49:39.743238 7041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:39.743238 7042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:39.743266 7044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:39.743913 7036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:39.744479 7036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:39.745639 7036 hybrid_clock.cc:648] HybridClock initialized: now 1790506179745622 us; error 30 us; skew 500 ppm
Sep 27 10:49:39 dist-test-slave-c60x krb5kdc[6795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:39.748597 7036 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:49:39.749531 7036 webserver.cc:460] Webserver started at http://127.6.31.130:44251/ using document root <none> and password file <none>
I20260927 10:49:39.750092 7036 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:39.750181 7036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:39.750408 7036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:39.752516 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "c2dfe054ecc542629e0d3c4b48800dd5"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "d14e3292b6b31bae04c5d5353889aff5"
server_key_iv: "d9ae70490382dac0c7a4707259adcb6f"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.753093 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "c2dfe054ecc542629e0d3c4b48800dd5"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "d14e3292b6b31bae04c5d5353889aff5"
server_key_iv: "d9ae70490382dac0c7a4707259adcb6f"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.756858 7036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:49:39.759231 7051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:39.760490 7036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:39.760614 7036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "c2dfe054ecc542629e0d3c4b48800dd5"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "d14e3292b6b31bae04c5d5353889aff5"
server_key_iv: "d9ae70490382dac0c7a4707259adcb6f"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.760775 7036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:39.782482 7036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:39.785766 7036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:39.785966 7036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:39.786552 7036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:39.787415 7036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:39.787468 7036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:39.787508 7036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:39.787555 7036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:39.796515 7036 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:36547
I20260927 10:49:39.796576 7164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:36547 every 8 connection(s)
I20260927 10:49:39.797418 7036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:49:39.799695 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 7036
I20260927 10:49:39.799860 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:49:39.800179 6270 external_mini_cluster.cc:1428] Setting key fb6418b89c9931842eefff1f12a385df
Sep 27 10:49:39 dist-test-slave-c60x krb5kdc[6795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:49:39.812258 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:39.799311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:41455 (local address 127.6.31.190:33909)
0927 10:49:39.799446 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:39.799450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:39.800433 (+ 983us) server_negotiation.cc:407] Connection header received
0927 10:49:39.802050 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:39.802053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:39.802097 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:39.802182 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:39.803716 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.804257 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.805433 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.805851 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.805978 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:39.808522 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:39.808529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:39.808534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:39.808569 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:39.810101 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:39.810760 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:39.810763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:39.810764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:39.810840 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:39.811182 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:39.811184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:39.811185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:39.811359 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:39.811431 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:39.811980 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 10:49:39.812081 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
I20260927 10:49:39.813344 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:39.799694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33909 (local address 127.6.31.130:41455)
0927 10:49:39.800247 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:39.800304 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:39.801813 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:39.802367 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:39.802379 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:39.802891 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:39.803571 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:39.803587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.804378 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.804381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:39.805276 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:39.805289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:39.806018 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:39.806020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:39.806163 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:39.806796 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:39.806820 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:39.808343 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:39.810203 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:39.810208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:39.810220 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:39.810493 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:39.810917 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:39.810919 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:39.810921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:39.811042 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:39.811469 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:39.811474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:39.811750 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 10:49:39.811901 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:49:39.812247 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":180,"threads_started":1}
I20260927 10:49:39.814602 7165 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33909
I20260927 10:49:39.814874 7165 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:39.815410 7165 heartbeater.cc:511] Master 127.6.31.190:33909 requested a full tablet report, sending...
I20260927 10:49:39.816618 6837 ts_manager.cc:194] Registered new tserver with Master: c2dfe054ecc542629e0d3c4b48800dd5 (127.6.31.130:36547)
I20260927 10:49:39.817543 6837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:41455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:49:39.830736 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33909
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:39.941830 7170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:39.942099 7170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:39.942145 7170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:39.946341 7170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:39.946405 7170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:39.946478 7170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:49:39.952059 7170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33909
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:39.953096 7170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:39.954051 7170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:39.963644 7178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:39.963649 7175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:39.963650 7176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:39.963949 7170 server_base.cc:1034] running on GCE node
I20260927 10:49:39.964372 7170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:39.965052 7170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:39.966264 7170 hybrid_clock.cc:648] HybridClock initialized: now 1790506179966231 us; error 36 us; skew 500 ppm
Sep 27 10:49:39 dist-test-slave-c60x krb5kdc[6795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:39.968967 7170 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:49:39.969846 7170 webserver.cc:460] Webserver started at http://127.6.31.131:36469/ using document root <none> and password file <none>
I20260927 10:49:39.970403 7170 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:39.970494 7170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:39.970718 7170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:39.972738 7170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "51a86d43fb0545efb7bd251b0eba37ad"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "ed87193f63203834702532893073ddfc"
server_key_iv: "102075f195a5f05f10c6eb85a9d8d65d"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.973291 7170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "51a86d43fb0545efb7bd251b0eba37ad"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "ed87193f63203834702532893073ddfc"
server_key_iv: "102075f195a5f05f10c6eb85a9d8d65d"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.977015 7170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 10:49:39.979738 7185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:39.981257 7170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:49:39.981402 7170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "51a86d43fb0545efb7bd251b0eba37ad"
format_stamp: "Formatted at 2026-09-27 10:49:39 on dist-test-slave-c60x"
server_key: "ed87193f63203834702532893073ddfc"
server_key_iv: "102075f195a5f05f10c6eb85a9d8d65d"
server_key_version: "encryptionkey@0"
I20260927 10:49:39.981518 7170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:39.995879 7170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:39.998886 7170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:39.999073 7170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:39.999698 7170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:40.000520 7170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:40.000587 7170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:40.000653 7170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:40.000699 7170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:40.009258 7170 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:36267
I20260927 10:49:40.009285 7298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:36267 every 8 connection(s)
I20260927 10:49:40.010142 7170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:49:40.017275 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 7170
I20260927 10:49:40.017400 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:49:40.017717 6270 external_mini_cluster.cc:1428] Setting key c7ad3315490a121e5a0f18a31a59f7d6
Sep 27 10:49:40 dist-test-slave-c60x krb5kdc[6795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:40.025743 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.012033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:52141 (local address 127.6.31.190:33909)
0927 10:49:40.012202 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.012207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.013001 (+ 794us) server_negotiation.cc:407] Connection header received
0927 10:49:40.014246 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.014248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.014302 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.014386 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:40.015584 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.016201 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.017719 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.018011 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.018105 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.021322 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:40.021328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:40.021333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:40.021366 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:40.023367 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.023973 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.023976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.023977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.024083 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.024532 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.024535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.024537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.024827 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:40.024932 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.025437 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.025562 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:49:40.026798 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.012431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33909 (local address 127.6.31.131:52141)
0927 10:49:40.012854 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:40.012896 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:40.014050 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:40.014533 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:40.014539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:40.014890 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:40.015446 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.015461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.016366 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.016374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.017534 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.017547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.018176 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.018181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.018368 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.019068 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:40.019117 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:40.021120 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:40.023495 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:40.023501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:40.023521 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:40.023860 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:40.024193 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:40.024197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:40.024200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:40.024425 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:40.024929 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:40.024935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:40.025182 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 10:49:40.025381 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 10:49:40.025699 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260927 10:49:40.028002 7299 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33909
I20260927 10:49:40.028280 7299 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:40.028784 7299 heartbeater.cc:511] Master 127.6.31.190:33909 requested a full tablet report, sending...
I20260927 10:49:40.029824 6837 ts_manager.cc:194] Registered new tserver with Master: 51a86d43fb0545efb7bd251b0eba37ad (127.6.31.131:36267)
I20260927 10:49:40.030493 6837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:52141
I20260927 10:49:40.031920 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:49:40.042163 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.035174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.6.31.190:33909)
0927 10:49:40.035315 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.035321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.035489 (+ 168us) server_negotiation.cc:407] Connection header received
0927 10:49:40.035582 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.035584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.035621 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.035739 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:40.036371 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.036889 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.037632 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.037869 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.037946 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.039207 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:40.039210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:40.039213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:40.039236 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:40.040736 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.041095 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.041098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.041099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.041167 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.041389 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.041391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.041392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.041584 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:40.041727 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.041920 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.042007 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:49:40.046770 6837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:49:40.050027 6837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:49:40.064505 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.059238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.6.31.130:36547)
0927 10:49:40.059623 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.059634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.059962 (+ 328us) server_negotiation.cc:407] Connection header received
0927 10:49:40.060040 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.060045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.060216 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.060390 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:40.060631 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.061362 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.062879 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.063781 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.063887 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.064023 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.064246 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.064341 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":109,"threads_started":1}
I20260927 10:49:40.064519 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.059057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:36547 (local address 127.0.0.1:39276)
0927 10:49:40.059762 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:40.059775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:40.059864 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:40.060382 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:40.060385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:40.060403 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:40.060499 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.060507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.061490 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.061493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.062743 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.062754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.063913 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.063918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.064034 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.064080 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 10:49:40.064209 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 10:49:40.064352 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260927 10:49:40.064976 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.058490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:37979 (local address 127.0.0.1:33302)
0927 10:49:40.059409 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:40.059424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:40.059509 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:40.060179 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:40.060187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:40.060214 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:40.060388 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.060396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.062030 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.062034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.063257 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.063268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.064542 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.064549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.064650 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.064681 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:49:40.064761 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:49:40.064839 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":65,"thread_start_us":110,"threads_started":1}
I20260927 10:49:40.065260 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.058310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:36267 (local address 127.0.0.1:58094)
0927 10:49:40.059147 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:40.059204 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:40.059338 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:40.060209 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:40.060212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:40.060225 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:40.060401 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.060408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.062063 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.062069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.063557 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.063568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.064791 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.064795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.064881 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.064903 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:49:40.064964 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 10:49:40.065119 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":189,"spinlock_wait_cycles":2432,"thread_start_us":94,"threads_started":1}
I20260927 10:49:40.065505 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.058633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.6.31.129:37979)
0927 10:49:40.059030 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.059046 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.059608 (+ 562us) server_negotiation.cc:407] Connection header received
0927 10:49:40.059685 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.059690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.059849 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.059980 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:40.060545 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.061878 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.063381 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.064238 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.064339 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.064474 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.065190 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.065294 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":79,"threads_started":1}
I20260927 10:49:40.065861 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.058443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.6.31.131:36267)
0927 10:49:40.059241 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.059253 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.059277 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:49:40.059552 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.059561 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.059792 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.059953 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:40.060558 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.061911 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.063696 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.064492 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.065462 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.065594 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.065641 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.065724 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":235,"threads_started":1}
I20260927 10:49:40.067881 7099 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebaf812f8984045aad543c316ad1aa4 (DEFAULT_TABLE table=test-table [id=bd810cfa94e040b9a8df698d9cfe7e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:40.068418 7233 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebaf812f8984045aad543c316ad1aa4 (DEFAULT_TABLE table=test-table [id=bd810cfa94e040b9a8df698d9cfe7e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:40.068506 6965 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebaf812f8984045aad543c316ad1aa4 (DEFAULT_TABLE table=test-table [id=bd810cfa94e040b9a8df698d9cfe7e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:40.068992 7099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebaf812f8984045aad543c316ad1aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:40.069386 7233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebaf812f8984045aad543c316ad1aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:40.069406 6965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebaf812f8984045aad543c316ad1aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:40.075896 7316 tablet_bootstrap.cc:492] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad: Bootstrap starting.
I20260927 10:49:40.076345 7317 tablet_bootstrap.cc:492] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347: Bootstrap starting.
I20260927 10:49:40.078150 7317 tablet_bootstrap.cc:654] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:40.078653 7316 tablet_bootstrap.cc:654] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:40.079319 7317 log.cc:824] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:40.079463 7318 tablet_bootstrap.cc:492] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5: Bootstrap starting.
I20260927 10:49:40.079496 7316 log.cc:824] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:40.081444 7316 tablet_bootstrap.cc:492] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad: No bootstrap required, opened a new log
I20260927 10:49:40.081693 7316 ts_tablet_manager.cc:1397] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:49:40.081897 7317 tablet_bootstrap.cc:492] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347: No bootstrap required, opened a new log
I20260927 10:49:40.082089 7317 ts_tablet_manager.cc:1397] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 10:49:40.082713 7318 tablet_bootstrap.cc:654] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:40.083618 7318 log.cc:824] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:40.085011 7316 raft_consensus.cc:348] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.085258 7316 raft_consensus.cc:374] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:40.085317 7316 raft_consensus.cc:729] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a86d43fb0545efb7bd251b0eba37ad, State: Initialized, Role: FOLLOWER
I20260927 10:49:40.085754 7318 tablet_bootstrap.cc:492] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5: No bootstrap required, opened a new log
I20260927 10:49:40.085747 7316 consensus_queue.cc:260] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.085954 7318 ts_tablet_manager.cc:1397] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:49:40.086375 7317 raft_consensus.cc:348] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.086647 7316 ts_tablet_manager.cc:1428] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:49:40.086655 7317 raft_consensus.cc:374] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:40.086735 7317 raft_consensus.cc:729] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b83d0b36934ff8973130bd1a5e8347, State: Initialized, Role: FOLLOWER
I20260927 10:49:40.087042 7299 heartbeater.cc:503] Master 127.6.31.190:33909 was elected leader, sending a full tablet report...
I20260927 10:49:40.087211 7317 consensus_queue.cc:260] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.087857 7031 heartbeater.cc:503] Master 127.6.31.190:33909 was elected leader, sending a full tablet report...
I20260927 10:49:40.088212 7317 ts_tablet_manager.cc:1428] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:49:40.090371 7318 raft_consensus.cc:348] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.090673 7318 raft_consensus.cc:374] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:40.090742 7318 raft_consensus.cc:729] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dfe054ecc542629e0d3c4b48800dd5, State: Initialized, Role: FOLLOWER
I20260927 10:49:40.091401 7318 consensus_queue.cc:260] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.092078 7165 heartbeater.cc:503] Master 127.6.31.190:33909 was elected leader, sending a full tablet report...
I20260927 10:49:40.092476 7318 ts_tablet_manager.cc:1428] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 10:49:40.261143 7300 tablet.cc:2369] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:49:40.295730 7322 raft_consensus.cc:482] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:40.295933 7322 raft_consensus.cc:504] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.297156 7322 leader_election.cc:288] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2dfe054ecc542629e0d3c4b48800dd5 (127.6.31.130:36547), 39b83d0b36934ff8973130bd1a5e8347 (127.6.31.129:37979)
W20260927 10:49:40.298571 7166 tablet.cc:2369] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:49:40.302410 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.297540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:57765 (local address 127.6.31.130:36547)
0927 10:49:40.297674 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.297679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.298097 (+ 418us) server_negotiation.cc:407] Connection header received
0927 10:49:40.298198 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.298203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.298262 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.298327 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:40.298786 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.299440 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.301134 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.301756 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.301875 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.301925 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.302199 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.302283 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:49:40.302441 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.297439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:36547 (local address 127.6.31.131:57765)
0927 10:49:40.297985 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:40.297999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:40.298110 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:40.298392 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:40.298398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:40.298420 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:40.298647 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.298655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.299671 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.299674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.301002 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.301008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.301896 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.301899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.302018 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.302041 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:49:40.302180 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 10:49:40.302261 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":176,"thread_start_us":223,"threads_started":1}
I20260927 10:49:40.303236 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.297439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:37979 (local address 127.6.31.131:34897)
0927 10:49:40.297541 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:40.297558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:40.297646 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:40.298280 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:40.298283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:40.298301 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:40.298485 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.298495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.299605 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.299608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.300998 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.301008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.302703 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.302708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.302840 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.302878 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:49:40.302965 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:49:40.303052 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":77}
I20260927 10:49:40.303239 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebaf812f8984045aad543c316ad1aa4" candidate_uuid: "51a86d43fb0545efb7bd251b0eba37ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" is_pre_election: true
I20260927 10:49:40.303429 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.297892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:34897 (local address 127.6.31.129:37979)
0927 10:49:40.298029 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.298033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.298044 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:49:40.298114 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.298117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.298172 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.298255 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:40.298624 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.299476 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.301980 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.302574 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.302684 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.302736 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.303197 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.303292 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:49:40.303546 7119 raft_consensus.cc:2454] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a86d43fb0545efb7bd251b0eba37ad in term 0.
I20260927 10:49:40.303973 6985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebaf812f8984045aad543c316ad1aa4" candidate_uuid: "51a86d43fb0545efb7bd251b0eba37ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b83d0b36934ff8973130bd1a5e8347" is_pre_election: true
I20260927 10:49:40.304023 7188 leader_election.cc:302] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a86d43fb0545efb7bd251b0eba37ad, c2dfe054ecc542629e0d3c4b48800dd5; no voters:
I20260927 10:49:40.304116 7323 raft_consensus.cc:482] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:40.304250 7323 raft_consensus.cc:504] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.304301 7322 raft_consensus.cc:2781] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:49:40.304376 7322 raft_consensus.cc:482] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:49:40.304415 7322 raft_consensus.cc:3037] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:40.304657 6985 raft_consensus.cc:2454] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a86d43fb0545efb7bd251b0eba37ad in term 0.
I20260927 10:49:40.305516 7323 leader_election.cc:288] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a86d43fb0545efb7bd251b0eba37ad (127.6.31.131:36267), c2dfe054ecc542629e0d3c4b48800dd5 (127.6.31.130:36547)
I20260927 10:49:40.305676 7322 raft_consensus.cc:504] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.306010 7322 leader_election.cc:288] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [CANDIDATE]: Term 1 election: Requested vote from peers c2dfe054ecc542629e0d3c4b48800dd5 (127.6.31.130:36547), 39b83d0b36934ff8973130bd1a5e8347 (127.6.31.129:37979)
I20260927 10:49:40.306386 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebaf812f8984045aad543c316ad1aa4" candidate_uuid: "51a86d43fb0545efb7bd251b0eba37ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dfe054ecc542629e0d3c4b48800dd5"
I20260927 10:49:40.306515 7119 raft_consensus.cc:3037] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:40.306964 6985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebaf812f8984045aad543c316ad1aa4" candidate_uuid: "51a86d43fb0545efb7bd251b0eba37ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b83d0b36934ff8973130bd1a5e8347"
I20260927 10:49:40.307068 6985 raft_consensus.cc:3037] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:40.308538 7119 raft_consensus.cc:2454] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a86d43fb0545efb7bd251b0eba37ad in term 1.
I20260927 10:49:40.308903 6985 raft_consensus.cc:2454] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a86d43fb0545efb7bd251b0eba37ad in term 1.
I20260927 10:49:40.308936 7188 leader_election.cc:302] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a86d43fb0545efb7bd251b0eba37ad, c2dfe054ecc542629e0d3c4b48800dd5; no voters:
I20260927 10:49:40.309140 7322 raft_consensus.cc:2781] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:40.309369 7322 raft_consensus.cc:686] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [term 1 LEADER]: Becoming Leader. State: Replica: 51a86d43fb0545efb7bd251b0eba37ad, State: Running, Role: LEADER
I20260927 10:49:40.309762 7322 consensus_queue.cc:237] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } }
I20260927 10:49:40.310760 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.305689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:36267 (local address 127.6.31.129:49957)
0927 10:49:40.306207 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:40.306222 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:40.306308 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:40.306817 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:40.306819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:40.306833 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:40.306945 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.306950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.307952 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.307958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.309192 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.309203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.310338 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.310341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.310427 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.310449 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:49:40.310544 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:49:40.310605 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":82,"threads_started":1}
I20260927 10:49:40.310773 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.305987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:49957 (local address 127.6.31.131:36267)
0927 10:49:40.306444 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.306449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.306463 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:49:40.306507 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.306510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.306560 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.306642 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:40.307058 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.307807 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.309320 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.309911 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.309989 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.310033 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.310585 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.310664 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370}
I20260927 10:49:40.311506 7253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebaf812f8984045aad543c316ad1aa4" candidate_uuid: "39b83d0b36934ff8973130bd1a5e8347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a86d43fb0545efb7bd251b0eba37ad" is_pre_election: true
I20260927 10:49:40.312158 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.308141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:60389 (local address 127.6.31.130:36547)
0927 10:49:40.308370 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.308374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.308388 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:49:40.308429 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.308432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.308470 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.308545 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:40.308918 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.309608 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.310976 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.311533 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.311620 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.311661 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.311952 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.312042 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260927 10:49:40.312496 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.306067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:36547 (local address 127.6.31.129:60389)
0927 10:49:40.306625 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:40.306636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:40.306719 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:40.308685 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:40.308689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:40.308707 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:40.308798 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.308803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.309777 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.309780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.310850 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:40.310860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.311685 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.311688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:40.311793 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.311817 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:49:40.312317 (+ 500us) client_negotiation.cc:240] Negotiation successful
0927 10:49:40.312391 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":107,"threads_started":1}
I20260927 10:49:40.312933 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebaf812f8984045aad543c316ad1aa4" candidate_uuid: "39b83d0b36934ff8973130bd1a5e8347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" is_pre_election: true
I20260927 10:49:40.313108 7119 raft_consensus.cc:2379] T 9ebaf812f8984045aad543c316ad1aa4 P c2dfe054ecc542629e0d3c4b48800dd5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39b83d0b36934ff8973130bd1a5e8347 in current term 1: Already voted for candidate 51a86d43fb0545efb7bd251b0eba37ad in this term.
I20260927 10:49:40.313558 6920 leader_election.cc:302] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39b83d0b36934ff8973130bd1a5e8347; no voters: 51a86d43fb0545efb7bd251b0eba37ad, c2dfe054ecc542629e0d3c4b48800dd5
I20260927 10:49:40.313499 6839 catalog_manager.cc:5495] T 9ebaf812f8984045aad543c316ad1aa4 P 51a86d43fb0545efb7bd251b0eba37ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 51a86d43fb0545efb7bd251b0eba37ad (127.6.31.131). New cstate: current_term: 1 leader_uuid: "51a86d43fb0545efb7bd251b0eba37ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86d43fb0545efb7bd251b0eba37ad" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2dfe054ecc542629e0d3c4b48800dd5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b83d0b36934ff8973130bd1a5e8347" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 37979 } health_report { overall_health: UNKNOWN } } }
I20260927 10:49:40.313948 7323 raft_consensus.cc:2726] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 10:49:40.326800 7032 tablet.cc:2369] T 9ebaf812f8984045aad543c316ad1aa4 P 39b83d0b36934ff8973130bd1a5e8347: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:49:40 dist-test-slave-c60x krb5kdc[6795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:49:40.360060 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.350798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.6.31.129:37979)
0927 10:49:40.350987 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.350992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.351151 (+ 159us) server_negotiation.cc:407] Connection header received
0927 10:49:40.351319 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.351323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.351377 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.351466 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:40.352162 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.352678 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.353534 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.353768 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.353879 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.356427 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:40.356438 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:40.356453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:40.356477 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:40.358147 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.358654 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.358660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.358661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.358743 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.359018 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.359023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.359025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.359342 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:40.359620 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.359770 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.359889 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 10:49:40 dist-test-slave-c60x krb5kdc[6795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506180, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:49:40 dist-test-slave-c60x krb5kdc[6795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506180, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:49:40.387603 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.379041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.6.31.129:37979)
0927 10:49:40.379226 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.379231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.379372 (+ 141us) server_negotiation.cc:407] Connection header received
0927 10:49:40.379539 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.379543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.379596 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.379689 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:40.380370 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.380896 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.381734 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.381978 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.382091 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.384555 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:40.384558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:40.384560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:40.384582 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:40.385829 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.386256 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.386259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.386263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.386337 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.386646 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.386649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.386651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.386815 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:40.386970 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.387306 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.387425 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 10:49:40 dist-test-slave-c60x krb5kdc[6795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:49:40 dist-test-slave-c60x krb5kdc[6795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:49:40.413048 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:40.404294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.6.31.129:37979)
0927 10:49:40.404478 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:40.404483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:40.404622 (+ 139us) server_negotiation.cc:407] Connection header received
0927 10:49:40.404773 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:40.404777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:40.404829 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:40.404931 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:40.405600 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.406144 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.406993 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:40.407280 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:40.407391 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:40.409932 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:40.409935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:40.409937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:40.409961 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:40.411372 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.411829 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.411834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.411836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.411916 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:40.412196 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:40.412200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:40.412202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:40.412374 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:40.412536 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:40.412754 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 10:49:40.412872 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 10:49:40.414369 6945 tablet_service.cc:2908] Rejecting scan request for tablet 9ebaf812f8984045aad543c316ad1aa4: Uninitialized: clean time has not yet been initialized
I20260927 10:49:40.416471 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 6902
I20260927 10:49:40.422484 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 7036
I20260927 10:49:40.428510 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 7170
I20260927 10:49:40.433933 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 6808
2026-09-27T10:49:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2486 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:40 dist-test-slave-c60x krb5kdc[7342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:40 dist-test-slave-c60x krb5kdc[7342](info): set up 2 sockets
Sep 27 10:49:40 dist-test-slave-c60x krb5kdc[7342](info): commencing operation
krb5kdc: starting...
W20260927 10:49:41.483299 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:49:41 dist-test-slave-c60x krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:49:41.611351 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:44399
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:44399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:41.722366 7355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:41.722625 7355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:41.722667 7355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:41.726619 7355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:41.726676 7355 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:49:41.726693 7355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:41.726711 7355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:41.726727 7355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:41.726743 7355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:41.731923 7355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:44399
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:44399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:41.732951 7355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:41.733922 7355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:41.742478 7363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:41.742473 7361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:41.742473 7360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:41.743052 7355 server_base.cc:1034] running on GCE node
I20260927 10:49:41.743635 7355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:41.744756 7355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:41.745931 7355 hybrid_clock.cc:648] HybridClock initialized: now 1790506181745911 us; error 30 us; skew 500 ppm
Sep 27 10:49:41 dist-test-slave-c60x krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506181, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:41.757387 7355 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:49:41.758519 7355 webserver.cc:460] Webserver started at http://127.6.31.190:34617/ using document root <none> and password file <none>
I20260927 10:49:41.759189 7355 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:41.759300 7355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:41.759521 7355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:41.761636 7355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "004c039350074db49c8838fbd2bb96f5"
format_stamp: "Formatted at 2026-09-27 10:49:41 on dist-test-slave-c60x"
server_key: "5695bd44212de073af71be5506aa7443"
server_key_iv: "f00b40492ac3c4f1d32a0e973ee67e39"
server_key_version: "encryptionkey@0"
I20260927 10:49:41.762213 7355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "004c039350074db49c8838fbd2bb96f5"
format_stamp: "Formatted at 2026-09-27 10:49:41 on dist-test-slave-c60x"
server_key: "5695bd44212de073af71be5506aa7443"
server_key_iv: "f00b40492ac3c4f1d32a0e973ee67e39"
server_key_version: "encryptionkey@0"
I20260927 10:49:41.766158 7355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:49:41.768698 7370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:41.769889 7355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:41.770037 7355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "004c039350074db49c8838fbd2bb96f5"
format_stamp: "Formatted at 2026-09-27 10:49:41 on dist-test-slave-c60x"
server_key: "5695bd44212de073af71be5506aa7443"
server_key_iv: "f00b40492ac3c4f1d32a0e973ee67e39"
server_key_version: "encryptionkey@0"
I20260927 10:49:41.770149 7355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:41.786419 7355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:41.789968 7355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:41.790205 7355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:41.798521 7355 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:44399
I20260927 10:49:41.798547 7432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:44399 every 8 connection(s)
I20260927 10:49:41.799502 7355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:41.802356 7433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:41.807763 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 7355
I20260927 10:49:41.807857 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:49:41.807881 7433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5: Bootstrap starting.
I20260927 10:49:41.808247 6270 external_mini_cluster.cc:1428] Setting key 7cbf976e0b07ca59855b947f2c805e69
I20260927 10:49:41.810654 7433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:41.811576 7433 log.cc:824] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:41.813983 7433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5: No bootstrap required, opened a new log
I20260927 10:49:41.818027 7433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004c039350074db49c8838fbd2bb96f5" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44399 } }
I20260927 10:49:41.818293 7433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:41.818367 7433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004c039350074db49c8838fbd2bb96f5, State: Initialized, Role: FOLLOWER
Sep 27 10:49:41 dist-test-slave-c60x krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:41.818812 7433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004c039350074db49c8838fbd2bb96f5" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44399 } }
I20260927 10:49:41.818930 7433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:41.819031 7433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:41.819151 7433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:41.820463 7433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004c039350074db49c8838fbd2bb96f5" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44399 } }
I20260927 10:49:41.820851 7433 leader_election.cc:302] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 004c039350074db49c8838fbd2bb96f5; no voters:
I20260927 10:49:41.821295 7433 leader_election.cc:288] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:41.821496 7438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:41.822141 7438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [term 1 LEADER]: Becoming Leader. State: Replica: 004c039350074db49c8838fbd2bb96f5, State: Running, Role: LEADER
I20260927 10:49:41.822544 7438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004c039350074db49c8838fbd2bb96f5" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44399 } }
I20260927 10:49:41.822682 7433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:41.824486 7439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "004c039350074db49c8838fbd2bb96f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004c039350074db49c8838fbd2bb96f5" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44399 } } }
I20260927 10:49:41.824613 7439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:41.824896 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:41.809721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.6.31.190:44399)
0927 10:49:41.810155 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:41.810174 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:41.810217 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:49:41.811176 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:41.811199 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:41.811610 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:41.811989 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:41.812866 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:41.814281 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:41.815405 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:41.816232 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:41.816411 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:41.819074 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:41.819105 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:41.819124 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:41.819162 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:41.821290 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:41.821888 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:41.821895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:41.821901 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:41.822048 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:41.822418 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:41.822422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:41.822424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:41.822951 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:41.823188 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:41.823475 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 10:49:41.823781 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":118,"threads_started":1}
I20260927 10:49:41.824890 7440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 004c039350074db49c8838fbd2bb96f5. Latest consensus state: current_term: 1 leader_uuid: "004c039350074db49c8838fbd2bb96f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004c039350074db49c8838fbd2bb96f5" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44399 } } }
I20260927 10:49:41.825353 7440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5 [sys.catalog]: This master's current role is: LEADER
W20260927 10:49:41.828035 7453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:41.828122 7453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:41.828193 7452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:41.828912 7452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:41.834633 7452 catalog_manager.cc:1347] Generated new cluster ID: 39fd0fe78900464a95cc10e7d1775f17
I20260927 10:49:41.834740 7452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:41.855417 7452 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:41.856805 7452 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:41.865856 7452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5: Generated new TSK 0
I20260927 10:49:41.866645 7452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:49:41.902688 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44399
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:42.014338 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:42.014655 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:42.014734 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:42.018776 7459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:42.018873 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:42.018946 7459 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:49:42.019024 7459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:49:42.027315 7459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.31.190:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:42.028865 7459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:42.029981 7459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:42.039891 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:42.039929 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:42.039988 7459 server_base.cc:1034] running on GCE node
W20260927 10:49:42.039937 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:42.040589 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:42.041270 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:42.042435 7459 hybrid_clock.cc:648] HybridClock initialized: now 1790506182042413 us; error 48 us; skew 500 ppm
Sep 27 10:49:42 dist-test-slave-c60x krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506182, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:42.045466 7459 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:49:42.046411 7459 webserver.cc:460] Webserver started at http://127.6.31.129:42085/ using document root <none> and password file <none>
I20260927 10:49:42.047009 7459 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:42.047072 7459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:42.047350 7459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:42.049453 7459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "968c0fdddfc145819892a2dd7813b94f"
format_stamp: "Formatted at 2026-09-27 10:49:42 on dist-test-slave-c60x"
server_key: "605c4c477090847a00228d1d89e1b6df"
server_key_iv: "94ac31d31d86d72fde7648a2b042e22c"
server_key_version: "encryptionkey@0"
I20260927 10:49:42.050020 7459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "968c0fdddfc145819892a2dd7813b94f"
format_stamp: "Formatted at 2026-09-27 10:49:42 on dist-test-slave-c60x"
server_key: "605c4c477090847a00228d1d89e1b6df"
server_key_iv: "94ac31d31d86d72fde7648a2b042e22c"
server_key_version: "encryptionkey@0"
I20260927 10:49:42.053875 7459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 10:49:42.056361 7474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:42.057639 7459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:42.057775 7459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "968c0fdddfc145819892a2dd7813b94f"
format_stamp: "Formatted at 2026-09-27 10:49:42 on dist-test-slave-c60x"
server_key: "605c4c477090847a00228d1d89e1b6df"
server_key_iv: "94ac31d31d86d72fde7648a2b042e22c"
server_key_version: "encryptionkey@0"
I20260927 10:49:42.057888 7459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:42.084501 7459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:42.088037 7459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:42.088258 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:42.089745 7459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:42.089818 7459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:42.089886 7459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:42.089921 7459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:49:42 dist-test-slave-c60x krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506182, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:42.100620 7459 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:40169
I20260927 10:49:42.100695 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:40169 every 8 connection(s)
I20260927 10:49:42.101697 7459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:49:42.103331 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.090695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:48091 (local address 127.6.31.190:44399)
0927 10:49:42.090845 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.090851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.091695 (+ 844us) server_negotiation.cc:407] Connection header received
0927 10:49:42.092730 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.092734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.092797 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.092900 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:42.094375 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.095263 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.096518 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.096951 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.097069 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.099593 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:42.099596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:42.099599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:42.099625 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:42.101315 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.101766 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.101770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.101771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.101860 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.102169 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.102174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.102176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.102404 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:42.102505 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.103037 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.103208 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:49:42.104100 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.091039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44399 (local address 127.6.31.129:48091)
0927 10:49:42.091521 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.091569 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.092489 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.093115 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.093123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.093555 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:42.094221 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.094253 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.095373 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.095377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.096369 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.096377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.097133 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.097140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.097359 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.097903 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:42.097918 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:42.099428 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:42.101419 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.101423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.101433 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.101665 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.101942 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.101944 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.101945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.102068 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.102555 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:42.102560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:42.102826 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.102974 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.103296 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":130,"threads_started":1}
I20260927 10:49:42.105506 7590 heartbeater.cc:347] Connected to a master server at 127.6.31.190:44399
I20260927 10:49:42.105655 7590 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:42.106213 7590 heartbeater.cc:511] Master 127.6.31.190:44399 requested a full tablet report, sending...
I20260927 10:49:42.107771 7386 ts_manager.cc:194] Registered new tserver with Master: 968c0fdddfc145819892a2dd7813b94f (127.6.31.129:40169)
I20260927 10:49:42.109275 7386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:48091
I20260927 10:49:42.109913 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 7459
I20260927 10:49:42.110037 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:49:42.110303 6270 external_mini_cluster.cc:1428] Setting key 4a76666d5abaae502a08a737a3cb9cf5
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:49:42.124010 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.115259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:49445 (local address 127.6.31.190:44399)
0927 10:49:42.115407 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.115414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.115433 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:49:42.115477 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.115483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.115536 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.115639 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:42.116176 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.117058 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.118506 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.118734 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.118829 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.120354 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:42.120366 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:42.120368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:42.120389 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:42.122015 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.122474 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.122477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.122478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.122558 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.122973 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.122976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.122977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.123241 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:42.123328 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.123648 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.123818 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:49:42.124226 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.114681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44399 (local address 127.6.31.129:49445)
0927 10:49:42.114960 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.114976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.115068 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.115668 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.115671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.115899 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:42.116029 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.116035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.117228 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.117232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.118354 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.118370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.118868 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.118873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.118992 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.119549 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:42.119567 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:42.120054 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:42.122117 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.122120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.122122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.122325 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.122668 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.122670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.122672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.122754 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.123341 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:42.123344 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:42.123426 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.123565 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.123735 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:49:42.142685 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44399
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:42.253945 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:42.254242 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:42.254320 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:42.258512 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:42.258603 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:42.258680 7599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:49:42.258764 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:49:42.264237 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.31.190:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:42.265403 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:42.266477 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:42.275974 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:42.275998 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:42.275977 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:42.276242 7599 server_base.cc:1034] running on GCE node
I20260927 10:49:42.276741 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:42.277505 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:42.278707 7599 hybrid_clock.cc:648] HybridClock initialized: now 1790506182278687 us; error 35 us; skew 500 ppm
Sep 27 10:49:42 dist-test-slave-c60x krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506182, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:42.282198 7599 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:49:42.283263 7599 webserver.cc:460] Webserver started at http://127.6.31.130:34647/ using document root <none> and password file <none>
I20260927 10:49:42.283818 7599 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:42.283879 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:42.284108 7599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:42.286288 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "b44cf15f4a4b4711a5b9863b649bc31c"
format_stamp: "Formatted at 2026-09-27 10:49:42 on dist-test-slave-c60x"
server_key: "033d8a7096f0cd3cb300e8335f30aeec"
server_key_iv: "9691111e6e4ded87a8ec2ad8b8eac241"
server_key_version: "encryptionkey@0"
I20260927 10:49:42.286901 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "b44cf15f4a4b4711a5b9863b649bc31c"
format_stamp: "Formatted at 2026-09-27 10:49:42 on dist-test-slave-c60x"
server_key: "033d8a7096f0cd3cb300e8335f30aeec"
server_key_iv: "9691111e6e4ded87a8ec2ad8b8eac241"
server_key_version: "encryptionkey@0"
I20260927 10:49:42.290865 7599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:49:42.293393 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:42.294625 7599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:42.294796 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "b44cf15f4a4b4711a5b9863b649bc31c"
format_stamp: "Formatted at 2026-09-27 10:49:42 on dist-test-slave-c60x"
server_key: "033d8a7096f0cd3cb300e8335f30aeec"
server_key_iv: "9691111e6e4ded87a8ec2ad8b8eac241"
server_key_version: "encryptionkey@0"
I20260927 10:49:42.294909 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:42.308431 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:42.311793 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:42.311997 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:42.313594 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:42.313643 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:42.313689 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:42.313753 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:49:42 dist-test-slave-c60x krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506182, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:42.324396 7599 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:40813
I20260927 10:49:42.324717 7729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:40813 every 8 connection(s)
I20260927 10:49:42.325510 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:49:42.328192 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.314297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:55685 (local address 127.6.31.190:44399)
0927 10:49:42.314461 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.314466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.315871 (+ 1405us) server_negotiation.cc:407] Connection header received
0927 10:49:42.316996 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.317000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.317058 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.317149 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:42.318824 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.319782 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.321008 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.321261 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.321380 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.324605 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:42.324608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:42.324612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:42.324638 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:42.326342 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.326841 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.326843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.326845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.326921 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.327266 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.327269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.327270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.327453 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:42.327538 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.327971 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.328070 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:49:42.328975 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.315219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44399 (local address 127.6.31.130:55685)
0927 10:49:42.315678 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.315727 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.316799 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.317348 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.317360 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.317873 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:42.318619 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.318641 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.319930 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.319936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.320878 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.320887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.321417 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.321421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.321604 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.322466 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:42.322490 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:42.324359 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:42.326463 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.326468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.326480 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.326749 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.326981 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.326983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.326984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.327160 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.327546 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:42.327551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:42.327766 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.327908 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.328146 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":139,"threads_started":1}
I20260927 10:49:42.329041 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 7599
I20260927 10:49:42.329140 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:49:42.329406 6270 external_mini_cluster.cc:1428] Setting key 2917a05abcdae716992ac219751a84c6
I20260927 10:49:42.330268 7730 heartbeater.cc:347] Connected to a master server at 127.6.31.190:44399
I20260927 10:49:42.330443 7730 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:42.330950 7730 heartbeater.cc:511] Master 127.6.31.190:44399 requested a full tablet report, sending...
I20260927 10:49:42.331966 7387 ts_manager.cc:194] Registered new tserver with Master: b44cf15f4a4b4711a5b9863b649bc31c (127.6.31.130:40813)
I20260927 10:49:42.332783 7387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:55685
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:49:42.340794 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.333085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:45179 (local address 127.6.31.190:44399)
0927 10:49:42.333207 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.333213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.333228 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:49:42.333264 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.333270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.333327 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.333412 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:42.333888 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.334669 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.335679 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.336097 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.336210 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.337372 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:42.337378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:42.337382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:42.337415 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:42.339055 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.339499 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.339505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.339507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.339593 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.339873 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.339878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.339879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.340098 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:42.340227 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.340512 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.340630 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:49:42.340802 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.332355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44399 (local address 127.6.31.130:45179)
0927 10:49:42.332606 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.332622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.332722 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.333429 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.333433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.333646 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:42.333752 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.333759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.334868 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.334871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.335550 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.335559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.336222 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.336225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.336295 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.336835 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:42.336846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:42.337214 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:42.339193 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.339196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.339198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.339391 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.339705 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.339707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.339709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.339782 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.340217 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:42.340220 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:42.340341 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.340459 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.340607 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:49:42.358868 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44399
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:42.469892 7739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:42.470227 7739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:42.470323 7739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:42.474977 7739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:42.475063 7739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:42.475158 7739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:49:42.475248 7739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:49:42.480763 7739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.31.190:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:42.481884 7739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:42.482903 7739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:42.492969 7744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:42.493046 7745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:42.492959 7739 server_base.cc:1034] running on GCE node
W20260927 10:49:42.492928 7747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:42.493506 7739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:42.494160 7739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:42.495348 7739 hybrid_clock.cc:648] HybridClock initialized: now 1790506182495333 us; error 27 us; skew 500 ppm
Sep 27 10:49:42 dist-test-slave-c60x krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506182, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:42.498229 7739 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:49:42.499282 7739 webserver.cc:460] Webserver started at http://127.6.31.131:36863/ using document root <none> and password file <none>
I20260927 10:49:42.499845 7739 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:42.499933 7739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:42.500165 7739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:42.502209 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "7868292db40b459fa2daaf830dc58f9d"
format_stamp: "Formatted at 2026-09-27 10:49:42 on dist-test-slave-c60x"
server_key: "a3b9d0f2c8b400a5383e1580efdd7512"
server_key_iv: "9035ca40255068755584bdaa212eee52"
server_key_version: "encryptionkey@0"
I20260927 10:49:42.502775 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "7868292db40b459fa2daaf830dc58f9d"
format_stamp: "Formatted at 2026-09-27 10:49:42 on dist-test-slave-c60x"
server_key: "a3b9d0f2c8b400a5383e1580efdd7512"
server_key_iv: "9035ca40255068755584bdaa212eee52"
server_key_version: "encryptionkey@0"
I20260927 10:49:42.506613 7739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:49:42.509179 7754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:42.510723 7739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 10:49:42.510859 7739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "7868292db40b459fa2daaf830dc58f9d"
format_stamp: "Formatted at 2026-09-27 10:49:42 on dist-test-slave-c60x"
server_key: "a3b9d0f2c8b400a5383e1580efdd7512"
server_key_iv: "9035ca40255068755584bdaa212eee52"
server_key_version: "encryptionkey@0"
I20260927 10:49:42.510957 7739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:42.530081 7739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:42.533773 7739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:42.533934 7739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:42.535391 7739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:42.535442 7739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:42.535508 7739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:42.535550 7739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:49:42 dist-test-slave-c60x krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506182, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:42.546316 7739 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:42307
I20260927 10:49:42.546320 7869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:42307 every 8 connection(s)
I20260927 10:49:42.547446 7739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:49:42.548792 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.536386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:47325 (local address 127.6.31.190:44399)
0927 10:49:42.536539 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.536546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.537365 (+ 819us) server_negotiation.cc:407] Connection header received
0927 10:49:42.538511 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.538516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.538584 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.538704 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:42.540072 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.540600 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.541727 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.541947 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.542064 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.544884 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:42.544889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:42.544892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:42.544923 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:42.546644 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.547126 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.547130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.547132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.547220 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.547542 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.547545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.547547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.547800 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:42.547914 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.548509 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.548629 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:49:42.549549 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.536750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44399 (local address 127.6.31.131:47325)
0927 10:49:42.537193 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.537242 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.538242 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.538909 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.538919 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.539350 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:42.539925 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.539939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.540727 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.540730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.541597 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.541606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.542112 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.542115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.542291 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.543041 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:42.543057 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:42.544630 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:42.546750 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.546755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.546765 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.547001 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.547315 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.547317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.547319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.547442 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.547954 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:42.547962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:42.548240 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.548463 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.548798 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":132,"threads_started":1}
I20260927 10:49:42.550783 7870 heartbeater.cc:347] Connected to a master server at 127.6.31.190:44399
I20260927 10:49:42.550925 7870 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:42.551513 7870 heartbeater.cc:511] Master 127.6.31.190:44399 requested a full tablet report, sending...
I20260927 10:49:42.552659 7386 ts_manager.cc:194] Registered new tserver with Master: 7868292db40b459fa2daaf830dc58f9d (127.6.31.131:42307)
I20260927 10:49:42.553433 7386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:47325
I20260927 10:49:42.555325 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 7739
I20260927 10:49:42.555446 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:49:42.555752 6270 external_mini_cluster.cc:1428] Setting key 8993fad8e29e2a8f12143faac5f75f38
I20260927 10:49:42.557715 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:49:42.561075 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.553210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:49543 (local address 127.6.31.190:44399)
0927 10:49:42.553338 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.553342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.553544 (+ 202us) server_negotiation.cc:407] Connection header received
0927 10:49:42.553681 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.553685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.553738 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.553811 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:42.554323 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.555117 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.556263 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.556555 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.556666 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.557842 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:42.557845 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:42.557848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:42.557871 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:42.559398 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.559876 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.559881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.559883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.559963 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.560265 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.560270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.560272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.560444 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:42.560552 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.560870 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.560967 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:49:42.561079 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.553153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44399 (local address 127.6.31.131:49543)
0927 10:49:42.553445 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.553463 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.553564 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.553843 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.553846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.554061 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:42.554204 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.554210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.555277 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.555306 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.556142 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.556151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.556669 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.556672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.556764 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.557360 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:42.557373 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:42.557688 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:42.559522 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.559525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.559528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.559780 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.560061 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.560064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.560065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.560149 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.560556 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:42.560560 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:42.560653 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.560804 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.560932 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":5,"thread_start_us":108,"threads_started":1}
Sep 27 10:49:42 dist-test-slave-c60x krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506182, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:49:42 dist-test-slave-c60x krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506182, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:42.584848 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.576572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.6.31.190:44399)
0927 10:49:42.576761 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.576765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.576777 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:49:42.576817 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.576819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.576858 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.576956 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:42.577689 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.578223 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.579049 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.579331 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.579444 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.581809 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:42.581813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:42.581815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:42.581839 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:42.583279 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.583661 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.583714 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.583716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.583792 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.584031 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.584033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.584034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.584205 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:42.584302 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.584565 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.584667 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 10:49:42.587915 7386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:49:42.590777 7386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:49:42.605538 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.599384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:40169 (local address 127.0.0.1:48558)
0927 10:49:42.600131 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.600188 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.600321 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.601192 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.601200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.601224 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:42.601419 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.601436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.602845 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.602848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.604062 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.604069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.604940 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.604943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.605048 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.605083 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.605223 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.605351 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":85,"spinlock_wait_cycles":6528,"thread_start_us":96,"threads_started":1}
I20260927 10:49:42.605520 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.600117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.6.31.129:40169)
0927 10:49:42.600725 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.600738 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.600756 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:49:42.600871 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.600876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.601035 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.601162 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:42.601596 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.602704 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.604171 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.604814 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.604939 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.605089 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.605286 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.605378 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":79,"threads_started":1}
I20260927 10:49:42.605818 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.599357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.6.31.130:40813)
0927 10:49:42.600550 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.600562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.600589 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:49:42.600666 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.600671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.600890 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.601051 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:42.601523 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.602306 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.603980 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.604835 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.604939 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.605548 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.605595 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.605689 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":99,"threads_started":1}
I20260927 10:49:42.605931 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.599819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.6.31.131:42307)
0927 10:49:42.600361 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.600374 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.600782 (+ 408us) server_negotiation.cc:407] Connection header received
0927 10:49:42.600864 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.600869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.601069 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.601269 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:42.601553 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.602719 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.604218 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.604955 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.605043 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.605178 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.605689 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.605770 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":114,"threads_started":1}
I20260927 10:49:42.606166 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.599355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:40813 (local address 127.0.0.1:40454)
0927 10:49:42.600138 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.600186 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.600321 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.601212 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.601215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.601233 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:42.601395 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.601402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.602425 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.602430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.603855 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.603866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.605141 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.605145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.605240 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.605263 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.605335 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.605418 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":26,"spinlock_wait_cycles":2176,"thread_start_us":72,"threads_started":1}
I20260927 10:49:42.605602 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.599610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:42307 (local address 127.0.0.1:36626)
0927 10:49:42.600560 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.600594 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.600682 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.601299 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.601302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.601314 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:42.601459 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.601467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.602860 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.602864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.604082 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.604093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.605227 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.605231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.605314 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.605342 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.605399 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.605450 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":126,"thread_start_us":101,"threads_started":1}
I20260927 10:49:42.608790 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9aa3fe0ead4f4eb9652cd2542630dd (DEFAULT_TABLE table=test-table [id=c9e8c065bbb34e7780193b2c3b51b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:42.608794 7804 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9aa3fe0ead4f4eb9652cd2542630dd (DEFAULT_TABLE table=test-table [id=c9e8c065bbb34e7780193b2c3b51b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:42.609731 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9aa3fe0ead4f4eb9652cd2542630dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:42.609738 7804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9aa3fe0ead4f4eb9652cd2542630dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:42.612627 7664 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9aa3fe0ead4f4eb9652cd2542630dd (DEFAULT_TABLE table=test-table [id=c9e8c065bbb34e7780193b2c3b51b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:42.613744 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9aa3fe0ead4f4eb9652cd2542630dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:42.616520 7894 tablet_bootstrap.cc:492] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: Bootstrap starting.
I20260927 10:49:42.616911 7893 tablet_bootstrap.cc:492] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: Bootstrap starting.
I20260927 10:49:42.619277 7893 tablet_bootstrap.cc:654] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:42.620137 7893 log.cc:824] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:42.620838 7896 tablet_bootstrap.cc:492] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: Bootstrap starting.
I20260927 10:49:42.621232 7894 tablet_bootstrap.cc:654] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:42.622068 7894 log.cc:824] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:42.622988 7896 tablet_bootstrap.cc:654] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:42.623245 7893 tablet_bootstrap.cc:492] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: No bootstrap required, opened a new log
I20260927 10:49:42.623435 7893 ts_tablet_manager.cc:1397] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:49:42.623873 7896 log.cc:824] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:42.624207 7894 tablet_bootstrap.cc:492] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: No bootstrap required, opened a new log
I20260927 10:49:42.624434 7894 ts_tablet_manager.cc:1397] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 10:49:42.625699 7896 tablet_bootstrap.cc:492] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: No bootstrap required, opened a new log
I20260927 10:49:42.625890 7896 ts_tablet_manager.cc:1397] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:49:42.627326 7893 raft_consensus.cc:348] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.627625 7893 raft_consensus.cc:374] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:42.627885 7893 raft_consensus.cc:729] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c0fdddfc145819892a2dd7813b94f, State: Initialized, Role: FOLLOWER
I20260927 10:49:42.628151 7894 raft_consensus.cc:348] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.628379 7894 raft_consensus.cc:374] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:42.628427 7894 raft_consensus.cc:729] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7868292db40b459fa2daaf830dc58f9d, State: Initialized, Role: FOLLOWER
I20260927 10:49:42.628391 7893 consensus_queue.cc:260] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.628855 7894 consensus_queue.cc:260] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.629338 7893 ts_tablet_manager.cc:1428] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:49:42.629256 7896 raft_consensus.cc:348] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.629491 7896 raft_consensus.cc:374] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:42.629612 7870 heartbeater.cc:503] Master 127.6.31.190:44399 was elected leader, sending a full tablet report...
I20260927 10:49:42.629640 7590 heartbeater.cc:503] Master 127.6.31.190:44399 was elected leader, sending a full tablet report...
I20260927 10:49:42.629788 7894 ts_tablet_manager.cc:1428] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:49:42.630184 7896 raft_consensus.cc:729] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44cf15f4a4b4711a5b9863b649bc31c, State: Initialized, Role: FOLLOWER
I20260927 10:49:42.630621 7896 consensus_queue.cc:260] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.631556 7896 ts_tablet_manager.cc:1428] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:49:42.631892 7730 heartbeater.cc:503] Master 127.6.31.190:44399 was elected leader, sending a full tablet report...
I20260927 10:49:42.686834 7899 raft_consensus.cc:482] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:42.687111 7899 raft_consensus.cc:504] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.688292 7899 leader_election.cc:288] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44cf15f4a4b4711a5b9863b649bc31c (127.6.31.130:40813), 7868292db40b459fa2daaf830dc58f9d (127.6.31.131:42307)
I20260927 10:49:42.692706 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.688749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:40777 (local address 127.6.31.131:42307)
0927 10:49:42.688909 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.688914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.689020 (+ 106us) server_negotiation.cc:407] Connection header received
0927 10:49:42.689163 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.689166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.689223 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.689344 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:42.689804 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.690401 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.691647 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.692139 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.692229 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.692268 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.692514 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.692579 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 10:49:42.692713 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.688556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:42307 (local address 127.6.31.129:40777)
0927 10:49:42.688907 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.688922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.689046 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.689423 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.689426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.689519 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:42.689678 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.689687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.690523 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.690526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.691528 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.691537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.692255 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.692258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.692358 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.692385 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.692483 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.692554 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"spinlock_wait_cycles":37760,"thread_start_us":111,"threads_started":1}
I20260927 10:49:42.693006 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.688700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:60785 (local address 127.6.31.130:40813)
0927 10:49:42.688909 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.688914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.689491 (+ 577us) server_negotiation.cc:407] Connection header received
0927 10:49:42.689614 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.689617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.689669 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.689763 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:42.690103 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.690782 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.691956 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.692420 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.692528 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.692569 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.692785 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.692867 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 10:49:42.693024 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.688556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:40813 (local address 127.6.31.129:60785)
0927 10:49:42.689377 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.689390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.689517 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.689814 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.689817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.689834 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:42.689941 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.689947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.690897 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.690899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.691830 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.691839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.692528 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.692532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.692631 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.692654 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.692732 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.692849 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":369,"spinlock_wait_cycles":12928,"thread_start_us":143,"threads_started":1}
I20260927 10:49:42.693477 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9aa3fe0ead4f4eb9652cd2542630dd" candidate_uuid: "968c0fdddfc145819892a2dd7813b94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7868292db40b459fa2daaf830dc58f9d" is_pre_election: true
I20260927 10:49:42.693493 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9aa3fe0ead4f4eb9652cd2542630dd" candidate_uuid: "968c0fdddfc145819892a2dd7813b94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" is_pre_election: true
I20260927 10:49:42.693786 7824 raft_consensus.cc:2454] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c0fdddfc145819892a2dd7813b94f in term 0.
I20260927 10:49:42.693796 7684 raft_consensus.cc:2454] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c0fdddfc145819892a2dd7813b94f in term 0.
I20260927 10:49:42.694303 7476 leader_election.cc:302] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7868292db40b459fa2daaf830dc58f9d, 968c0fdddfc145819892a2dd7813b94f, b44cf15f4a4b4711a5b9863b649bc31c; no voters:
I20260927 10:49:42.694600 7899 raft_consensus.cc:2781] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:49:42.694684 7899 raft_consensus.cc:482] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:49:42.694727 7899 raft_consensus.cc:3037] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:42.695989 7899 raft_consensus.cc:504] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.696355 7899 leader_election.cc:288] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [CANDIDATE]: Term 1 election: Requested vote from peers b44cf15f4a4b4711a5b9863b649bc31c (127.6.31.130:40813), 7868292db40b459fa2daaf830dc58f9d (127.6.31.131:42307)
I20260927 10:49:42.696653 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9aa3fe0ead4f4eb9652cd2542630dd" candidate_uuid: "968c0fdddfc145819892a2dd7813b94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44cf15f4a4b4711a5b9863b649bc31c"
I20260927 10:49:42.696712 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9aa3fe0ead4f4eb9652cd2542630dd" candidate_uuid: "968c0fdddfc145819892a2dd7813b94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7868292db40b459fa2daaf830dc58f9d"
I20260927 10:49:42.696832 7824 raft_consensus.cc:3037] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:42.696830 7684 raft_consensus.cc:3037] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:42.697921 7824 raft_consensus.cc:2454] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c0fdddfc145819892a2dd7813b94f in term 1.
I20260927 10:49:42.698276 7478 leader_election.cc:302] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7868292db40b459fa2daaf830dc58f9d, 968c0fdddfc145819892a2dd7813b94f; no voters:
I20260927 10:49:42.698458 7899 raft_consensus.cc:2781] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:42.698448 7684 raft_consensus.cc:2454] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c0fdddfc145819892a2dd7813b94f in term 1.
I20260927 10:49:42.698675 7899 raft_consensus.cc:686] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 1 LEADER]: Becoming Leader. State: Replica: 968c0fdddfc145819892a2dd7813b94f, State: Running, Role: LEADER
I20260927 10:49:42.699002 7899 consensus_queue.cc:237] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.701689 7386 catalog_manager.cc:5495] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c0fdddfc145819892a2dd7813b94f (127.6.31.129). New cstate: current_term: 1 leader_uuid: "968c0fdddfc145819892a2dd7813b94f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260927 10:49:42.732612 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.728643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.6.31.129:40169)
0927 10:49:42.728779 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.728788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.728832 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:49:42.729021 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.729023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.729077 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.729156 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:42.729432 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.729976 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.731078 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.731361 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.731438 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.731765 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:42.732159 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:42.732250 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.732382 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.732450 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:49:42.738077 7824 raft_consensus.cc:1261] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Refusing update from remote peer 968c0fdddfc145819892a2dd7813b94f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:42.738077 7684 raft_consensus.cc:1261] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [term 1 FOLLOWER]: Refusing update from remote peer 968c0fdddfc145819892a2dd7813b94f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:42.738765 7904 consensus_queue.cc:1035] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:42.738958 7899 consensus_queue.cc:1035] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:42.745443 7907 mvcc.cc:205] Tried to move back new op lower bound from 7333913324488450048 to 7333913324337799168. Current Snapshot: MvccSnapshot[applied={T|T < 7333913324488450048}]
I20260927 10:49:42.747674 7909 mvcc.cc:205] Tried to move back new op lower bound from 7333913324488450048 to 7333913324337799168. Current Snapshot: MvccSnapshot[applied={T|T < 7333913324488450048}]
I20260927 10:49:42.759044 7386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:42.759330 7386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:42.761972 7386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5: Sending DeleteTablet for 3 replicas of tablet 9d9aa3fe0ead4f4eb9652cd2542630dd
I20260927 10:49:42.762849 7664 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9aa3fe0ead4f4eb9652cd2542630dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:42 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:40454
I20260927 10:49:42.762845 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9aa3fe0ead4f4eb9652cd2542630dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:42 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:48558
I20260927 10:49:42.763495 7804 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9aa3fe0ead4f4eb9652cd2542630dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:42 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:36626
I20260927 10:49:42.763748 7916 tablet_replica.cc:330] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: stopping tablet replica
I20260927 10:49:42.763835 7386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:49:42.764091 7916 raft_consensus.cc:2229] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:49:42.764206 7918 tablet_replica.cc:330] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: stopping tablet replica
W20260927 10:49:42.764278 7386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:49:42.764503 7918 raft_consensus.cc:2229] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:42.764511 7916 raft_consensus.cc:2258] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:42.764945 7918 raft_consensus.cc:2258] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:42.765033 7917 tablet_replica.cc:330] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: stopping tablet replica
I20260927 10:49:42.765347 7917 raft_consensus.cc:2229] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:42.765662 7917 raft_consensus.cc:2258] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:42.767584 7918 ts_tablet_manager.cc:1905] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:42.767648 7916 ts_tablet_manager.cc:1905] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:42.770339 7918 ts_tablet_manager.cc:1918] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:49:42.770468 7918 log.cc:1196] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/9d9aa3fe0ead4f4eb9652cd2542630dd
I20260927 10:49:42.770504 7916 ts_tablet_manager.cc:1918] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:49:42.770612 7916 log.cc:1196] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/9d9aa3fe0ead4f4eb9652cd2542630dd
I20260927 10:49:42.770838 7918 ts_tablet_manager.cc:1939] T 9d9aa3fe0ead4f4eb9652cd2542630dd P b44cf15f4a4b4711a5b9863b649bc31c: Deleting consensus metadata
I20260927 10:49:42.770949 7916 ts_tablet_manager.cc:1939] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 968c0fdddfc145819892a2dd7813b94f: Deleting consensus metadata
I20260927 10:49:42.771251 7664 tablet_service.cc:1467] Processing CreateTablet for tablet 988314c148924bef9444aa5e871634dc (DEFAULT_TABLE table=test-table [id=4fd78f310e074ac9a43fb54d11639fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:42.771502 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988314c148924bef9444aa5e871634dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:42.772272 7371 catalog_manager.cc:4841] TS 968c0fdddfc145819892a2dd7813b94f (127.6.31.129:40169): tablet 9d9aa3fe0ead4f4eb9652cd2542630dd (table test-table [id=c9e8c065bbb34e7780193b2c3b51b5cf]) successfully deleted
I20260927 10:49:42.773106 7804 tablet_service.cc:1467] Processing CreateTablet for tablet 988314c148924bef9444aa5e871634dc (DEFAULT_TABLE table=test-table [id=4fd78f310e074ac9a43fb54d11639fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:42.773384 7804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988314c148924bef9444aa5e871634dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:42.773679 7896 tablet_bootstrap.cc:492] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: Bootstrap starting.
I20260927 10:49:42.774592 7896 tablet_bootstrap.cc:654] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:42.775197 7917 ts_tablet_manager.cc:1905] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:42.775240 7372 catalog_manager.cc:4841] TS b44cf15f4a4b4711a5b9863b649bc31c (127.6.31.130:40813): tablet 9d9aa3fe0ead4f4eb9652cd2542630dd (table test-table [id=c9e8c065bbb34e7780193b2c3b51b5cf]) successfully deleted
I20260927 10:49:42.775677 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 988314c148924bef9444aa5e871634dc (DEFAULT_TABLE table=test-table [id=4fd78f310e074ac9a43fb54d11639fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:42.775936 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988314c148924bef9444aa5e871634dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:42.775974 7894 tablet_bootstrap.cc:492] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: Bootstrap starting.
I20260927 10:49:42.776871 7894 tablet_bootstrap.cc:654] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:42.777922 7917 ts_tablet_manager.cc:1918] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:49:42.778060 7917 log.cc:1196] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/9d9aa3fe0ead4f4eb9652cd2542630dd
I20260927 10:49:42.778316 7893 tablet_bootstrap.cc:492] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: Bootstrap starting.
I20260927 10:49:42.778342 7894 tablet_bootstrap.cc:492] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: No bootstrap required, opened a new log
I20260927 10:49:42.778417 7894 ts_tablet_manager.cc:1397] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:42.778553 7917 ts_tablet_manager.cc:1939] T 9d9aa3fe0ead4f4eb9652cd2542630dd P 7868292db40b459fa2daaf830dc58f9d: Deleting consensus metadata
I20260927 10:49:42.778882 7894 raft_consensus.cc:348] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.779008 7894 raft_consensus.cc:374] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:42.779048 7894 raft_consensus.cc:729] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7868292db40b459fa2daaf830dc58f9d, State: Initialized, Role: FOLLOWER
I20260927 10:49:42.779217 7894 consensus_queue.cc:260] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.779343 7893 tablet_bootstrap.cc:654] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:42.779515 7894 ts_tablet_manager.cc:1428] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:49:42.779724 7896 tablet_bootstrap.cc:492] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: No bootstrap required, opened a new log
I20260927 10:49:42.779815 7896 ts_tablet_manager.cc:1397] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260927 10:49:42.780284 7896 raft_consensus.cc:348] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.780346 7374 catalog_manager.cc:4841] TS 7868292db40b459fa2daaf830dc58f9d (127.6.31.131:42307): tablet 9d9aa3fe0ead4f4eb9652cd2542630dd (table test-table [id=c9e8c065bbb34e7780193b2c3b51b5cf]) successfully deleted
I20260927 10:49:42.780409 7896 raft_consensus.cc:374] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:42.780442 7896 raft_consensus.cc:729] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44cf15f4a4b4711a5b9863b649bc31c, State: Initialized, Role: FOLLOWER
I20260927 10:49:42.780603 7896 consensus_queue.cc:260] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.780864 7896 ts_tablet_manager.cc:1428] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:49:42.782632 7893 tablet_bootstrap.cc:492] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: No bootstrap required, opened a new log
I20260927 10:49:42.782716 7893 ts_tablet_manager.cc:1397] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 10:49:42.783177 7893 raft_consensus.cc:348] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.783296 7893 raft_consensus.cc:374] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:42.783329 7893 raft_consensus.cc:729] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c0fdddfc145819892a2dd7813b94f, State: Initialized, Role: FOLLOWER
I20260927 10:49:42.783447 7893 consensus_queue.cc:260] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.783704 7893 ts_tablet_manager.cc:1428] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 10:49:42.798664 7871 tablet.cc:2369] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:49:42.822254 7901 raft_consensus.cc:482] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:42.822414 7901 raft_consensus.cc:504] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.823756 7901 leader_election.cc:288] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968c0fdddfc145819892a2dd7813b94f (127.6.31.129:40169), 7868292db40b459fa2daaf830dc58f9d (127.6.31.131:42307)
W20260927 10:49:42.826808 7731 tablet.cc:2369] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:49:42.828140 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.823990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:40169 (local address 127.6.31.130:60643)
0927 10:49:42.824248 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.824264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.824391 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.824682 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.824686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.824717 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:42.824858 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.824865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.825682 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.825688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.826842 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.826852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.827704 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.827710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.827824 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.827850 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.827926 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.827996 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260927 10:49:42.828606 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.824157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:47493 (local address 127.6.31.131:42307)
0927 10:49:42.824269 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.824275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.824779 (+ 504us) server_negotiation.cc:407] Connection header received
0927 10:49:42.824881 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.824884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.824926 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.825003 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:42.825240 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.825981 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.827391 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.827924 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.828011 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.828062 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.828406 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.828475 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:49:42.828734 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.824149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:60643 (local address 127.6.31.129:40169)
0927 10:49:42.824267 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.824271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.824331 (+ 60us) server_negotiation.cc:407] Connection header received
0927 10:49:42.824510 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.824513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.824569 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.824658 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:42.824959 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.825577 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.826987 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.827585 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.827721 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.827777 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.828542 (+ 765us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.828618 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:49:42.828945 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.823990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:42307 (local address 127.6.31.130:47493)
0927 10:49:42.824698 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.824711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.824797 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.825012 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.825014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.825030 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:42.825132 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.825137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.826100 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.826103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.827264 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.827273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.828157 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.828160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.828271 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.828292 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.828737 (+ 445us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.828810 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":294,"thread_start_us":96,"threads_started":1}
I20260927 10:49:42.829190 7530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988314c148924bef9444aa5e871634dc" candidate_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c0fdddfc145819892a2dd7813b94f" is_pre_election: true
I20260927 10:49:42.829349 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988314c148924bef9444aa5e871634dc" candidate_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7868292db40b459fa2daaf830dc58f9d" is_pre_election: true
I20260927 10:49:42.829409 7530 raft_consensus.cc:2454] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b44cf15f4a4b4711a5b9863b649bc31c in term 0.
I20260927 10:49:42.829463 7824 raft_consensus.cc:2454] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b44cf15f4a4b4711a5b9863b649bc31c in term 0.
I20260927 10:49:42.829816 7618 leader_election.cc:302] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7868292db40b459fa2daaf830dc58f9d, b44cf15f4a4b4711a5b9863b649bc31c; no voters:
I20260927 10:49:42.830103 7901 raft_consensus.cc:2781] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:49:42.830171 7901 raft_consensus.cc:482] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:49:42.830231 7901 raft_consensus.cc:3037] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:42.831465 7901 raft_consensus.cc:504] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.831849 7901 leader_election.cc:288] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [CANDIDATE]: Term 1 election: Requested vote from peers 968c0fdddfc145819892a2dd7813b94f (127.6.31.129:40169), 7868292db40b459fa2daaf830dc58f9d (127.6.31.131:42307)
I20260927 10:49:42.832182 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988314c148924bef9444aa5e871634dc" candidate_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7868292db40b459fa2daaf830dc58f9d"
I20260927 10:49:42.832288 7824 raft_consensus.cc:3037] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:42.833282 7824 raft_consensus.cc:2454] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b44cf15f4a4b4711a5b9863b649bc31c in term 1.
I20260927 10:49:42.833551 7530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988314c148924bef9444aa5e871634dc" candidate_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c0fdddfc145819892a2dd7813b94f"
I20260927 10:49:42.833598 7618 leader_election.cc:302] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7868292db40b459fa2daaf830dc58f9d, b44cf15f4a4b4711a5b9863b649bc31c; no voters:
I20260927 10:49:42.833680 7530 raft_consensus.cc:3037] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:42.833760 7901 raft_consensus.cc:2781] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:42.833923 7901 raft_consensus.cc:686] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 1 LEADER]: Becoming Leader. State: Replica: b44cf15f4a4b4711a5b9863b649bc31c, State: Running, Role: LEADER
I20260927 10:49:42.834398 7901 consensus_queue.cc:237] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.834831 7530 raft_consensus.cc:2454] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b44cf15f4a4b4711a5b9863b649bc31c in term 1.
I20260927 10:49:42.836860 7386 catalog_manager.cc:5495] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c reported cstate change: term changed from 0 to 1, leader changed from <none> to b44cf15f4a4b4711a5b9863b649bc31c (127.6.31.130). New cstate: current_term: 1 leader_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } health_report { overall_health: UNKNOWN } } }
W20260927 10:49:42.853231 7591 tablet.cc:2369] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:49:42 dist-test-slave-c60x krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506181, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:42.879689 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.869127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.6.31.190:44399)
0927 10:49:42.869282 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.869287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.869505 (+ 218us) server_negotiation.cc:407] Connection header received
0927 10:49:42.869700 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.869703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.869745 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.869819 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:42.870716 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.871325 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.872511 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.872729 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.872812 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.876052 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:42.876055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:42.876057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:42.876078 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:42.877617 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.878260 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.878264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.878265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.878340 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:42.878651 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:42.878653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:42.878654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:42.878867 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:42.878945 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.879319 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.879538 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":67}
I20260927 10:49:42.879699 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.869061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44399 (local address 127.0.0.1:53254)
0927 10:49:42.869392 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.869409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.869538 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.869873 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.869878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.870384 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:42.870512 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.870520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.871437 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.871440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.872361 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.872376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.872855 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.872861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.872978 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.873780 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:42.873803 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:42.875888 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:42.877752 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.877769 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.877779 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.878147 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.878438 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:42.878441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:42.878443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:42.878554 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:42.878989 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:42.878998 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:42.879155 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.879277 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.879488 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":1,"thread_start_us":128,"threads_started":1}
I20260927 10:49:42.884022 7386 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:53254:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:49:42.884872 7386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:49:42.890202 7664 tablet_service.cc:1467] Processing CreateTablet for tablet 182e9cc35ffa4c3986683ff3bf3ba073 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89b9885c30e14bec9173a131d295a12d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:49:42.890172 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 182e9cc35ffa4c3986683ff3bf3ba073 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89b9885c30e14bec9173a131d295a12d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:49:42.890610 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182e9cc35ffa4c3986683ff3bf3ba073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:42.890604 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182e9cc35ffa4c3986683ff3bf3ba073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:42.893123 7896 tablet_bootstrap.cc:492] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: Bootstrap starting.
I20260927 10:49:42.893553 7804 tablet_service.cc:1467] Processing CreateTablet for tablet 182e9cc35ffa4c3986683ff3bf3ba073 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89b9885c30e14bec9173a131d295a12d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:49:42.893913 7893 tablet_bootstrap.cc:492] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: Bootstrap starting.
I20260927 10:49:42.894295 7804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182e9cc35ffa4c3986683ff3bf3ba073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:42.894491 7896 tablet_bootstrap.cc:654] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:42.894833 7893 tablet_bootstrap.cc:654] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:42.896144 7893 tablet_bootstrap.cc:492] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: No bootstrap required, opened a new log
I20260927 10:49:42.896234 7893 ts_tablet_manager.cc:1397] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:42.896740 7893 raft_consensus.cc:348] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.896878 7893 raft_consensus.cc:374] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:42.896911 7893 raft_consensus.cc:729] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c0fdddfc145819892a2dd7813b94f, State: Initialized, Role: FOLLOWER
I20260927 10:49:42.897032 7893 consensus_queue.cc:260] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.897312 7893 ts_tablet_manager.cc:1428] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:42.897523 7894 tablet_bootstrap.cc:492] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: Bootstrap starting.
I20260927 10:49:42.898175 7904 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.898334 7904 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:42.898524 7894 tablet_bootstrap.cc:654] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:42.899847 7894 tablet_bootstrap.cc:492] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: No bootstrap required, opened a new log
I20260927 10:49:42.899931 7894 ts_tablet_manager.cc:1397] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 10:49:42.900092 7896 tablet_bootstrap.cc:492] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: No bootstrap required, opened a new log
I20260927 10:49:42.900178 7896 ts_tablet_manager.cc:1397] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260927 10:49:42.900377 7894 raft_consensus.cc:348] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.900472 7894 raft_consensus.cc:374] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:42.900511 7894 raft_consensus.cc:729] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7868292db40b459fa2daaf830dc58f9d, State: Initialized, Role: FOLLOWER
I20260927 10:49:42.900597 7896 raft_consensus.cc:348] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.900637 7894 consensus_queue.cc:260] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.900700 7896 raft_consensus.cc:374] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:42.900732 7896 raft_consensus.cc:729] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44cf15f4a4b4711a5b9863b649bc31c, State: Initialized, Role: FOLLOWER
I20260927 10:49:42.900844 7896 consensus_queue.cc:260] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.900918 7894 ts_tablet_manager.cc:1428] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:49:42.901103 7896 ts_tablet_manager.cc:1428] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:42.901381 7900 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.901530 7900 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:42.901679 7901 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.901772 7901 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:42.946842 7904 raft_consensus.cc:482] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:42.946978 7904 raft_consensus.cc:504] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.947459 7904 leader_election.cc:288] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44cf15f4a4b4711a5b9863b649bc31c (127.6.31.130:40813), 7868292db40b459fa2daaf830dc58f9d (127.6.31.131:42307)
I20260927 10:49:42.947934 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182e9cc35ffa4c3986683ff3bf3ba073" candidate_uuid: "968c0fdddfc145819892a2dd7813b94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" is_pre_election: true
I20260927 10:49:42.947934 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182e9cc35ffa4c3986683ff3bf3ba073" candidate_uuid: "968c0fdddfc145819892a2dd7813b94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7868292db40b459fa2daaf830dc58f9d" is_pre_election: true
I20260927 10:49:42.948096 7684 raft_consensus.cc:2454] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c0fdddfc145819892a2dd7813b94f in term 0.
I20260927 10:49:42.948096 7824 raft_consensus.cc:2454] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c0fdddfc145819892a2dd7813b94f in term 0.
I20260927 10:49:42.948395 7478 leader_election.cc:302] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7868292db40b459fa2daaf830dc58f9d, 968c0fdddfc145819892a2dd7813b94f; no voters:
I20260927 10:49:42.948565 7904 raft_consensus.cc:2781] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:49:42.948622 7904 raft_consensus.cc:482] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:49:42.948649 7904 raft_consensus.cc:3037] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:42.949671 7904 raft_consensus.cc:504] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.950027 7904 leader_election.cc:288] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [CANDIDATE]: Term 1 election: Requested vote from peers b44cf15f4a4b4711a5b9863b649bc31c (127.6.31.130:40813), 7868292db40b459fa2daaf830dc58f9d (127.6.31.131:42307)
I20260927 10:49:42.950369 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182e9cc35ffa4c3986683ff3bf3ba073" candidate_uuid: "968c0fdddfc145819892a2dd7813b94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7868292db40b459fa2daaf830dc58f9d"
I20260927 10:49:42.950376 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182e9cc35ffa4c3986683ff3bf3ba073" candidate_uuid: "968c0fdddfc145819892a2dd7813b94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44cf15f4a4b4711a5b9863b649bc31c"
I20260927 10:49:42.950471 7824 raft_consensus.cc:3037] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:42.950507 7684 raft_consensus.cc:3037] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:42.951475 7824 raft_consensus.cc:2454] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c0fdddfc145819892a2dd7813b94f in term 1.
I20260927 10:49:42.951476 7684 raft_consensus.cc:2454] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c0fdddfc145819892a2dd7813b94f in term 1.
I20260927 10:49:42.951752 7478 leader_election.cc:302] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7868292db40b459fa2daaf830dc58f9d, 968c0fdddfc145819892a2dd7813b94f; no voters:
I20260927 10:49:42.951913 7904 raft_consensus.cc:2781] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:42.952049 7904 raft_consensus.cc:686] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [term 1 LEADER]: Becoming Leader. State: Replica: 968c0fdddfc145819892a2dd7813b94f, State: Running, Role: LEADER
I20260927 10:49:42.952190 7904 consensus_queue.cc:237] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } }
I20260927 10:49:42.952900 7899 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: TxnStatusTablet state changed. Reason: New leader 968c0fdddfc145819892a2dd7813b94f. Latest consensus state: current_term: 1 leader_uuid: "968c0fdddfc145819892a2dd7813b94f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.953008 7899 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:49:42.953269 7935 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:49:42.953442 7935 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:49:42.953965 7387 catalog_manager.cc:5495] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c0fdddfc145819892a2dd7813b94f (127.6.31.129). New cstate: current_term: 1 leader_uuid: "968c0fdddfc145819892a2dd7813b94f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260927 10:49:42.967523 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.963855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.6.31.129:40169)
0927 10:49:42.963973 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.963977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.963993 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:49:42.964145 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.964148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.964199 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.964287 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:42.964513 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.965337 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.966202 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.966570 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.966665 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.966992 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:42.967114 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:42.967188 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.967333 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.967386 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:49:42.967581 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.963783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:40169 (local address 127.0.0.1:48586)
0927 10:49:42.963922 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.963944 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.964032 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.964308 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.964310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.964317 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:49:42.964415 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.964419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.965460 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.965463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.966077 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.966085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.966741 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.966745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.966816 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.966862 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:49:42.967213 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:49:42.967230 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.967320 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.967411 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 10:49:42.969942 7824 raft_consensus.cc:1261] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Refusing update from remote peer 968c0fdddfc145819892a2dd7813b94f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:42.970067 7684 raft_consensus.cc:1261] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c [term 1 FOLLOWER]: Refusing update from remote peer 968c0fdddfc145819892a2dd7813b94f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:42.970358 7899 consensus_queue.cc:1035] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:42.970498 7904 consensus_queue.cc:1035] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:42.971853 7901 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: TxnStatusTablet state changed. Reason: New leader 968c0fdddfc145819892a2dd7813b94f. Latest consensus state: current_term: 1 leader_uuid: "968c0fdddfc145819892a2dd7813b94f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.971872 7900 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: TxnStatusTablet state changed. Reason: New leader 968c0fdddfc145819892a2dd7813b94f. Latest consensus state: current_term: 1 leader_uuid: "968c0fdddfc145819892a2dd7813b94f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.971990 7901 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:42.971992 7900 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:42.974610 7899 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "968c0fdddfc145819892a2dd7813b94f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.974722 7899 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:49:42.974826 7899 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "968c0fdddfc145819892a2dd7813b94f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.974895 7899 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P 968c0fdddfc145819892a2dd7813b94f: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:49:42.975484 7901 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "968c0fdddfc145819892a2dd7813b94f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.975579 7901 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P b44cf15f4a4b4711a5b9863b649bc31c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:42.976660 7908 mvcc.cc:205] Tried to move back new op lower bound from 7333913325440692224 to 7333913325373554688. Current Snapshot: MvccSnapshot[applied={T|T < 7333913325440692224}]
I20260927 10:49:42.976666 7900 tablet_replica.cc:439] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "968c0fdddfc145819892a2dd7813b94f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 } } peers { permanent_uuid: "b44cf15f4a4b4711a5b9863b649bc31c" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 40813 } } peers { permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 } } }
I20260927 10:49:42.977097 7900 tablet_replica.cc:442] T 182e9cc35ffa4c3986683ff3bf3ba073 P 7868292db40b459fa2daaf830dc58f9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:42.984135 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.980405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.6.31.130:40813)
0927 10:49:42.980517 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.980522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.980539 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:49:42.980710 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.980712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.980763 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.980845 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:42.981094 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.981594 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.982714 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.982929 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.983008 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.983303 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:42.983735 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:42.983828 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.983924 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.983970 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:49:42.995121 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.991458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:54455 (local address 127.6.31.129:40169)
0927 10:49:42.991587 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:42.991591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:42.991710 (+ 119us) server_negotiation.cc:407] Connection header received
0927 10:49:42.991911 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:42.991914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:42.991966 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:42.992045 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:42.992633 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.993131 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.994015 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.994247 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.994360 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.994628 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:42.994705 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:42.994778 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:42.994907 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 10:49:42.994959 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:49:42.995143 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:42.991322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:40169 (local address 127.6.31.130:54455)
0927 10:49:42.991613 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:42.991631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:42.991765 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:42.992068 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:42.992071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:42.992079 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:49:42.992519 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.992525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.993235 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.993240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.993890 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:42.993898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:42.994357 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:42.994360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:42.994456 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:42.994496 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:49:42.994792 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:49:42.994805 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 10:49:42.994885 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:49:42.994964 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
I20260927 10:49:43.002030 7530 raft_consensus.cc:1261] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [term 1 FOLLOWER]: Refusing update from remote peer b44cf15f4a4b4711a5b9863b649bc31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:43.002092 7824 raft_consensus.cc:1261] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Refusing update from remote peer b44cf15f4a4b4711a5b9863b649bc31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:43.002555 7923 consensus_queue.cc:1035] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c0fdddfc145819892a2dd7813b94f" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:43.002748 7901 consensus_queue.cc:1035] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7868292db40b459fa2daaf830dc58f9d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:43.025947 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:43.022114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:46907 (local address 127.6.31.130:40813)
0927 10:49:43.022236 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:43.022241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:43.022454 (+ 213us) server_negotiation.cc:407] Connection header received
0927 10:49:43.022567 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:43.022570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:43.022628 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:43.022734 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:43.023033 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:43.023881 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:43.024781 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:43.025115 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:43.025228 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:43.025495 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:43.025576 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:43.025647 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:43.025755 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 10:49:43.025807 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:49:43.025995 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:43.021995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:40813 (local address 127.6.31.129:46907)
0927 10:49:43.022363 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:43.022379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:43.022485 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:43.022756 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:43.022759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:43.022768 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:49:43.022885 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:43.022891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:43.023988 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:43.023991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:43.024663 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:43.024672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:43.025225 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:43.025228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:43.025320 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:43.025365 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:49:43.025676 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:49:43.025682 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:49:43.025747 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 10:49:43.025831 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":119,"threads_started":1}
I20260927 10:49:43.052474 7386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:43.052666 7386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:43.055112 7386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 004c039350074db49c8838fbd2bb96f5: Sending DeleteTablet for 3 replicas of tablet 988314c148924bef9444aa5e871634dc
I20260927 10:49:43.055832 7804 tablet_service.cc:1514] Processing DeleteTablet for tablet 988314c148924bef9444aa5e871634dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:36626
I20260927 10:49:43.056010 7917 tablet_replica.cc:330] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: stopping tablet replica
I20260927 10:49:43.056113 7917 raft_consensus.cc:2229] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:43.056097 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet 988314c148924bef9444aa5e871634dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:48558
I20260927 10:49:43.056216 7917 raft_consensus.cc:2258] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:43.056381 7916 tablet_replica.cc:330] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: stopping tablet replica
I20260927 10:49:43.056488 7916 raft_consensus.cc:2229] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:43.056581 7916 raft_consensus.cc:2258] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:43.057642 7916 ts_tablet_manager.cc:1905] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:43.058369 7917 ts_tablet_manager.cc:1905] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:43.059299 7664 tablet_service.cc:1514] Processing DeleteTablet for tablet 988314c148924bef9444aa5e871634dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:40454
I20260927 10:49:43.059468 7918 tablet_replica.cc:330] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: stopping tablet replica
I20260927 10:49:43.059581 7918 raft_consensus.cc:2229] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:49:43.059851 7918 raft_consensus.cc:2258] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:43.060626 7918 ts_tablet_manager.cc:1905] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:43.061242 7917 ts_tablet_manager.cc:1918] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:49:43.061465 7917 log.cc:1196] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/988314c148924bef9444aa5e871634dc
I20260927 10:49:43.061738 7917 ts_tablet_manager.cc:1939] T 988314c148924bef9444aa5e871634dc P 7868292db40b459fa2daaf830dc58f9d: Deleting consensus metadata
I20260927 10:49:43.062515 7374 catalog_manager.cc:4841] TS 7868292db40b459fa2daaf830dc58f9d (127.6.31.131:42307): tablet 988314c148924bef9444aa5e871634dc (table test-table [id=4fd78f310e074ac9a43fb54d11639fa4]) successfully deleted
I20260927 10:49:43.063153 7918 ts_tablet_manager.cc:1918] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:49:43.063220 7918 log.cc:1196] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/988314c148924bef9444aa5e871634dc
I20260927 10:49:43.063472 7918 ts_tablet_manager.cc:1939] T 988314c148924bef9444aa5e871634dc P b44cf15f4a4b4711a5b9863b649bc31c: Deleting consensus metadata
I20260927 10:49:43.064142 7372 catalog_manager.cc:4841] TS b44cf15f4a4b4711a5b9863b649bc31c (127.6.31.130:40813): tablet 988314c148924bef9444aa5e871634dc (table test-table [id=4fd78f310e074ac9a43fb54d11639fa4]) successfully deleted
I20260927 10:49:43.065742 7916 ts_tablet_manager.cc:1918] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:49:43.065821 7916 log.cc:1196] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/988314c148924bef9444aa5e871634dc
I20260927 10:49:43.066052 7916 ts_tablet_manager.cc:1939] T 988314c148924bef9444aa5e871634dc P 968c0fdddfc145819892a2dd7813b94f: Deleting consensus metadata
I20260927 10:49:43.066867 7371 catalog_manager.cc:4841] TS 968c0fdddfc145819892a2dd7813b94f (127.6.31.129:40169): tablet 988314c148924bef9444aa5e871634dc (table test-table [id=4fd78f310e074ac9a43fb54d11639fa4]) successfully deleted
Sep 27 10:49:43 dist-test-slave-c60x krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506182, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:49:43.079188 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:43.064205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.6.31.129:40169)
0927 10:49:43.064346 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:43.064351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:43.066661 (+ 2310us) server_negotiation.cc:407] Connection header received
0927 10:49:43.066994 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:43.066998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:43.067046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:43.067118 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:43.068098 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:43.068905 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:43.070135 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:43.070489 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:43.070606 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:43.074622 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:43.074635 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:43.074654 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:43.074686 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:43.077000 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:43.077531 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:43.077538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:43.077540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:43.077631 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:43.077924 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:43.077928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:43.077930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:43.078244 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:43.078427 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:43.078811 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 10:49:43.078944 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:49:43.079774 7584 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48594
I20260927 10:49:43.091060 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:43.081643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.6.31.190:44399)
0927 10:49:43.081775 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:43.081780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:43.081996 (+ 216us) server_negotiation.cc:407] Connection header received
0927 10:49:43.082099 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:43.082102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:43.082150 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:43.082240 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:43.082967 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:43.083852 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:43.084916 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:43.085267 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:43.085369 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:43.087052 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:43.087056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:43.087059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:43.087139 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:43.088952 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:43.089467 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:43.089470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:43.089472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:43.089556 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:43.090120 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:43.090124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:43.090126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:43.090352 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:43.090482 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:43.090766 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 10:49:43.090888 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 10:49:43.091498 7386 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53260
I20260927 10:49:43.093685 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 7459
I20260927 10:49:43.102375 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 7599
I20260927 10:49:43.108522 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 7739
I20260927 10:49:43.114346 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 7355
2026-09-27T10:49:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2680 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:43 dist-test-slave-c60x krb5kdc[7956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:43 dist-test-slave-c60x krb5kdc[7956](info): set up 2 sockets
Sep 27 10:49:43 dist-test-slave-c60x krb5kdc[7956](info): commencing operation
krb5kdc: starting...
W20260927 10:49:44.158967 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:49:44 dist-test-slave-c60x krb5kdc[7956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:49:44.287578 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33397
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:33397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:44.399420 7969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:44.399762 7969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:44.399850 7969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:44.403952 7969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:44.404050 7969 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:49:44.404080 7969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:44.404124 7969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:44.404156 7969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:44.404191 7969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:44.410193 7969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:33397
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:44.411405 7969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:44.412506 7969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:44.421289 7975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:44.421289 7974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:44.421293 7977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:44.421700 7969 server_base.cc:1034] running on GCE node
I20260927 10:49:44.422331 7969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:44.423643 7969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:44.424970 7969 hybrid_clock.cc:648] HybridClock initialized: now 1790506184424937 us; error 50 us; skew 500 ppm
Sep 27 10:49:44 dist-test-slave-c60x krb5kdc[7956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506184, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:44.428509 7969 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:49:44.429466 7969 webserver.cc:460] Webserver started at http://127.6.31.190:40719/ using document root <none> and password file <none>
I20260927 10:49:44.430074 7969 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:44.430177 7969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:44.430426 7969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:44.432677 7969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "d3b7bf511b404287a0724eaf96fa595f"
format_stamp: "Formatted at 2026-09-27 10:49:44 on dist-test-slave-c60x"
server_key: "086ab3675fedec99c7bf1bf75b199fb4"
server_key_iv: "c20a822eba24d9945f9ab8d2d7579953"
server_key_version: "encryptionkey@0"
I20260927 10:49:44.433236 7969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "d3b7bf511b404287a0724eaf96fa595f"
format_stamp: "Formatted at 2026-09-27 10:49:44 on dist-test-slave-c60x"
server_key: "086ab3675fedec99c7bf1bf75b199fb4"
server_key_iv: "c20a822eba24d9945f9ab8d2d7579953"
server_key_version: "encryptionkey@0"
I20260927 10:49:44.437381 7969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:49:44.440796 7984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:44.442106 7969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:49:44.442260 7969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "d3b7bf511b404287a0724eaf96fa595f"
format_stamp: "Formatted at 2026-09-27 10:49:44 on dist-test-slave-c60x"
server_key: "086ab3675fedec99c7bf1bf75b199fb4"
server_key_iv: "c20a822eba24d9945f9ab8d2d7579953"
server_key_version: "encryptionkey@0"
I20260927 10:49:44.442389 7969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:44.468891 7969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:44.472535 7969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:44.472751 7969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:44.481251 7969 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:33397
I20260927 10:49:44.481266 8046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:33397 every 8 connection(s)
I20260927 10:49:44.482246 7969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:44.484552 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 7969
I20260927 10:49:44.484648 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:49:44.484980 6270 external_mini_cluster.cc:1428] Setting key 2240994d75c7c6b3ed9531dd7133b59e
I20260927 10:49:44.485306 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:44.491640 8047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f: Bootstrap starting.
Sep 27 10:49:44 dist-test-slave-c60x krb5kdc[7956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:44.493952 8047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:44.494825 8047 log.cc:824] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:44.496968 8047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f: No bootstrap required, opened a new log
I20260927 10:49:44.499568 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:44.486451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.6.31.190:33397)
0927 10:49:44.486910 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:44.486926 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:44.486962 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:49:44.487713 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:44.487730 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:44.488056 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:44.488379 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:44.489159 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.490136 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.490985 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.491546 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.491713 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:44.494137 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:44.494142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:44.494154 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:44.494181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:44.496600 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:44.497091 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:44.497100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:44.497111 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:44.497236 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:44.497535 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:44.497540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:44.497541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:44.497976 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:44.498180 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:44.498470 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 10:49:44.498761 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20260927 10:49:44.500262 8047 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7bf511b404287a0724eaf96fa595f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33397 } }
I20260927 10:49:44.500506 8047 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:44.500559 8047 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b7bf511b404287a0724eaf96fa595f, State: Initialized, Role: FOLLOWER
I20260927 10:49:44.501021 8047 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7bf511b404287a0724eaf96fa595f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33397 } }
I20260927 10:49:44.501175 8047 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:44.501262 8047 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:44.501382 8047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:44.502421 8047 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7bf511b404287a0724eaf96fa595f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33397 } }
I20260927 10:49:44.502768 8047 leader_election.cc:302] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3b7bf511b404287a0724eaf96fa595f; no voters:
I20260927 10:49:44.503072 8047 leader_election.cc:288] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:44.503248 8052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:44.503504 8052 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [term 1 LEADER]: Becoming Leader. State: Replica: d3b7bf511b404287a0724eaf96fa595f, State: Running, Role: LEADER
I20260927 10:49:44.503796 8052 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7bf511b404287a0724eaf96fa595f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33397 } }
I20260927 10:49:44.504388 8047 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:44.505522 8053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3b7bf511b404287a0724eaf96fa595f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7bf511b404287a0724eaf96fa595f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33397 } } }
I20260927 10:49:44.505751 8054 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3b7bf511b404287a0724eaf96fa595f. Latest consensus state: current_term: 1 leader_uuid: "d3b7bf511b404287a0724eaf96fa595f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7bf511b404287a0724eaf96fa595f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33397 } } }
I20260927 10:49:44.505867 8054 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:44.505748 8053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:44.506724 8061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:44.509684 8061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:44.515677 8061 catalog_manager.cc:1347] Generated new cluster ID: dd80a21c6b4a4c96a3c00105d0ba3d3c
I20260927 10:49:44.515756 8061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:44.529531 8061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:44.530725 8061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:44.535573 8061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f: Generated new TSK 0
I20260927 10:49:44.536352 8061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:49:44.572160 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33397
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:44.690152 8073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:44.690409 8073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:44.690517 8073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:44.694614 8073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:44.694684 8073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:44.694748 8073 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:49:44.694801 8073 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:49:44.694855 8073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:49:44.700220 8073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.31.190:33397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:44.701306 8073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:44.702382 8073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:44.712687 8079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:44.712793 8078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:44.712677 8081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:44.712971 8073 server_base.cc:1034] running on GCE node
I20260927 10:49:44.713418 8073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:44.714083 8073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:44.715273 8073 hybrid_clock.cc:648] HybridClock initialized: now 1790506184715264 us; error 31 us; skew 500 ppm
Sep 27 10:49:44 dist-test-slave-c60x krb5kdc[7956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506184, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:44.718171 8073 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:49:44.719251 8073 webserver.cc:460] Webserver started at http://127.6.31.129:46653/ using document root <none> and password file <none>
I20260927 10:49:44.719816 8073 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:44.719877 8073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:44.720085 8073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:44.722067 8073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "b5445a4527d64988ab038632451dff78"
format_stamp: "Formatted at 2026-09-27 10:49:44 on dist-test-slave-c60x"
server_key: "83484d1fa1ea5745ddff691c177d9f05"
server_key_iv: "6dc1d147147c12b0ab6beb7887e9e6ca"
server_key_version: "encryptionkey@0"
I20260927 10:49:44.722609 8073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "b5445a4527d64988ab038632451dff78"
format_stamp: "Formatted at 2026-09-27 10:49:44 on dist-test-slave-c60x"
server_key: "83484d1fa1ea5745ddff691c177d9f05"
server_key_iv: "6dc1d147147c12b0ab6beb7887e9e6ca"
server_key_version: "encryptionkey@0"
I20260927 10:49:44.726280 8073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:49:44.728794 8088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:44.730062 8073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:44.730189 8073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "b5445a4527d64988ab038632451dff78"
format_stamp: "Formatted at 2026-09-27 10:49:44 on dist-test-slave-c60x"
server_key: "83484d1fa1ea5745ddff691c177d9f05"
server_key_iv: "6dc1d147147c12b0ab6beb7887e9e6ca"
server_key_version: "encryptionkey@0"
I20260927 10:49:44.730294 8073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:44.757892 8073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:44.761360 8073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:44.761574 8073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:44.762997 8073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:44.763048 8073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:44.763140 8073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:44.763180 8073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:49:44 dist-test-slave-c60x krb5kdc[7956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506184, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:44.774248 8073 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:42713
I20260927 10:49:44.774323 8203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:42713 every 8 connection(s)
I20260927 10:49:44.775230 8073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:49:44.777079 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:44.764042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:52747 (local address 127.6.31.190:33397)
0927 10:49:44.764199 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:44.764205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:44.765073 (+ 868us) server_negotiation.cc:407] Connection header received
0927 10:49:44.766360 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:44.766365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:44.766433 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:44.766539 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:44.767985 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.768532 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.769936 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.770241 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.770360 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:44.773043 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:44.773046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:44.773048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:44.773072 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:44.774701 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:44.775323 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:44.775327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:44.775329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:44.775424 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:44.775994 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:44.775997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:44.775999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:44.776249 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:44.776366 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:44.776852 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 10:49:44.776961 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:49:44.777784 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:44.764453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33397 (local address 127.6.31.129:52747)
0927 10:49:44.764897 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:44.764945 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:44.766101 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:44.766757 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:44.766763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:44.767220 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:44.767834 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:44.767848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.768664 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.768668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:44.769812 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:44.769821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.770407 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.770411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:44.770573 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:44.771280 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:44.771304 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:44.772865 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:44.774822 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:44.774828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:44.774843 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:44.775188 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:44.775554 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:44.775587 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:44.775593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:44.775807 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:44.776351 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:44.776356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:44.776561 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 10:49:44.776718 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 10:49:44.776952 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":127,"threads_started":1}
I20260927 10:49:44.779158 8204 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33397
I20260927 10:49:44.779313 8204 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:44.779345 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 8073
I20260927 10:49:44.779461 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:49:44.779760 6270 external_mini_cluster.cc:1428] Setting key a96267358bc07d6ff7d543363d57b52f
I20260927 10:49:44.779928 8204 heartbeater.cc:511] Master 127.6.31.190:33397 requested a full tablet report, sending...
I20260927 10:49:44.782248 8000 ts_manager.cc:194] Registered new tserver with Master: b5445a4527d64988ab038632451dff78 (127.6.31.129:42713)
I20260927 10:49:44.783854 8000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:52747
I20260927 10:49:44.790347 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:44.781661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33397 (local address 127.6.31.129:56499)
0927 10:49:44.781937 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:44.781954 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:44.782056 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:44.782324 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:44.782329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:44.782562 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:44.782800 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:44.782810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.783996 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.784002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:44.784774 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:44.784783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.785512 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.785516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:44.785637 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:44.786264 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:44.786281 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:44.786617 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:44.788852 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:44.788854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:44.788856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:44.789051 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:44.789369 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:44.789371 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:44.789373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:44.789446 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:44.789892 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:44.789895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:44.789969 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 10:49:44.790056 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:49:44.790161 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
I20260927 10:49:44.790516 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:44.781583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:56499 (local address 127.6.31.190:33397)
0927 10:49:44.781720 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:44.781725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:44.782044 (+ 319us) server_negotiation.cc:407] Connection header received
0927 10:49:44.782152 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:44.782156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:44.782202 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:44.782291 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:44.782944 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.783833 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.784903 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.785362 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.785504 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:44.786773 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:44.786781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:44.786788 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:44.786822 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:44.788729 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:44.789154 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:44.789158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:44.789161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:44.789262 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:44.789537 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:44.789541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:44.789545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:44.789786 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:44.789901 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:44.790154 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 10:49:44.790289 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:49:44.810871 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33397
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:44.923940 8213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:44.924244 8213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:44.924321 8213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:44.928439 8213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:44.928529 8213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:44.928599 8213 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:49:44.928642 8213 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:49:44.928716 8213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:49:44.934129 8213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.31.190:33397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:44.935340 8213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:44.936378 8213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:44.945775 8219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:44.945817 8221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:44.945785 8218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:44.945981 8213 server_base.cc:1034] running on GCE node
I20260927 10:49:44.946415 8213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:44.947041 8213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:44.948235 8213 hybrid_clock.cc:648] HybridClock initialized: now 1790506184948199 us; error 55 us; skew 500 ppm
Sep 27 10:49:44 dist-test-slave-c60x krb5kdc[7956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506184, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:44.950999 8213 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:49:44.951853 8213 webserver.cc:460] Webserver started at http://127.6.31.130:46673/ using document root <none> and password file <none>
I20260927 10:49:44.952396 8213 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:44.952454 8213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:44.952683 8213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:44.954653 8213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "a2300a0bbf8a4cffa903cf48f8083400"
format_stamp: "Formatted at 2026-09-27 10:49:44 on dist-test-slave-c60x"
server_key: "86dd1514055cdc9acb145ff57eff8861"
server_key_iv: "430928570d89b0639cf1fd58accd710f"
server_key_version: "encryptionkey@0"
I20260927 10:49:44.955268 8213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "a2300a0bbf8a4cffa903cf48f8083400"
format_stamp: "Formatted at 2026-09-27 10:49:44 on dist-test-slave-c60x"
server_key: "86dd1514055cdc9acb145ff57eff8861"
server_key_iv: "430928570d89b0639cf1fd58accd710f"
server_key_version: "encryptionkey@0"
I20260927 10:49:44.958917 8213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:49:44.961299 8228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:44.962486 8213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:44.962622 8213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "a2300a0bbf8a4cffa903cf48f8083400"
format_stamp: "Formatted at 2026-09-27 10:49:44 on dist-test-slave-c60x"
server_key: "86dd1514055cdc9acb145ff57eff8861"
server_key_iv: "430928570d89b0639cf1fd58accd710f"
server_key_version: "encryptionkey@0"
I20260927 10:49:44.962730 8213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:44.974324 8213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:44.977660 8213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:44.977881 8213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:44.979400 8213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:44.979472 8213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:44.979552 8213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:44.979600 8213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:49:44 dist-test-slave-c60x krb5kdc[7956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506184, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:44.991004 8213 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:46539
I20260927 10:49:44.991438 8343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:46539 every 8 connection(s)
I20260927 10:49:44.992215 8213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:49:44.996349 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:44.980510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:38467 (local address 127.6.31.190:33397)
0927 10:49:44.980640 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:44.980644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:44.981646 (+ 1002us) server_negotiation.cc:407] Connection header received
0927 10:49:44.982842 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:44.982849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:44.982921 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:44.983025 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:44.984239 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.984766 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.986021 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.986363 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.986496 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:44.990277 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:44.990284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:44.990291 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:44.990361 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:44.993117 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:44.993891 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:44.993896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:44.993898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:44.994002 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:44.994495 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:44.994499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:44.994502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:44.994784 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:44.994898 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:44.996056 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0927 10:49:44.996179 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:49:44.996920 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:44.981012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33397 (local address 127.6.31.130:38467)
0927 10:49:44.981496 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:44.981537 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:44.982595 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:44.983234 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:44.983246 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:44.983626 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:44.984086 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:44.984098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.984903 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.984906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:44.985875 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:44.985892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:44.986542 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:44.986546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:44.986772 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:44.987841 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:44.987863 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:44.990049 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:44.993269 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:44.993276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:44.993292 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:44.993726 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:44.994117 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:44.994122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:44.994125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:44.994351 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:44.994907 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:44.994913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:44.995285 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 10:49:44.995513 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 10:49:44.995960 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":146,"threads_started":1}
I20260927 10:49:44.997824 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 8213
I20260927 10:49:44.997956 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:49:44.998248 6270 external_mini_cluster.cc:1428] Setting key acf73f3e2f76f6b0e13e75df54d5a24b
I20260927 10:49:44.998292 8344 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33397
I20260927 10:49:44.998427 8344 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:44.999042 8344 heartbeater.cc:511] Master 127.6.31.190:33397 requested a full tablet report, sending...
I20260927 10:49:45.000097 8001 ts_manager.cc:194] Registered new tserver with Master: a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130:46539)
I20260927 10:49:45.000697 8001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:38467
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:49:45.010555 8351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.001588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33397 (local address 127.6.31.130:52107)
0927 10:49:45.001949 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.001963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.002081 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.002312 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.002314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.002574 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:45.002745 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.002756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.003615 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.003622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.004682 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.004695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.005427 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.005433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.005567 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.006220 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:45.006235 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:45.006901 (+ 666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:45.008982 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:45.008985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:45.008988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:45.009245 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:45.009541 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:45.009543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:45.009545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:45.009622 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:45.010055 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:45.010061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:45.010162 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.010293 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.010411 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":205,"threads_started":1}
I20260927 10:49:45.010571 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.001625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:52107 (local address 127.6.31.190:33397)
0927 10:49:45.001722 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.001728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.002042 (+ 314us) server_negotiation.cc:407] Connection header received
0927 10:49:45.002158 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.002161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.002200 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.002296 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:45.002878 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.003463 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.004904 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.005297 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.005391 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.007100 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:45.007106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:45.007110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:45.007139 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:45.008830 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.009352 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.009355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.009357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.009448 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.009702 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.009704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.009705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.009975 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:45.010055 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.010351 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.010456 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:49:45.029723 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33397
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:45.146878 8353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:45.147189 8353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:45.147243 8353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:45.151293 8353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:45.151348 8353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:45.151389 8353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:49:45.151403 8353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:49:45.151436 8353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:49:45.156677 8353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.31.190:33397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:45.157670 8353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:45.158627 8353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:45.168227 8358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:45.168255 8361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:45.168442 8353 server_base.cc:1034] running on GCE node
W20260927 10:49:45.168452 8359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:45.168963 8353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:45.169528 8353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:45.170689 8353 hybrid_clock.cc:648] HybridClock initialized: now 1790506185170674 us; error 25 us; skew 500 ppm
Sep 27 10:49:45 dist-test-slave-c60x krb5kdc[7956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506185, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:45.173511 8353 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:49:45.174437 8353 webserver.cc:460] Webserver started at http://127.6.31.131:37665/ using document root <none> and password file <none>
I20260927 10:49:45.174993 8353 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:45.175081 8353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:45.175329 8353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:45.177300 8353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "2fe85a810a1b4c718f64b1d60debc2f8"
format_stamp: "Formatted at 2026-09-27 10:49:45 on dist-test-slave-c60x"
server_key: "b8ec08f6e3d9b8cc92bd63f424f27af0"
server_key_iv: "1e045c1c41e712af9b4ac67f06771cdb"
server_key_version: "encryptionkey@0"
I20260927 10:49:45.177878 8353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "2fe85a810a1b4c718f64b1d60debc2f8"
format_stamp: "Formatted at 2026-09-27 10:49:45 on dist-test-slave-c60x"
server_key: "b8ec08f6e3d9b8cc92bd63f424f27af0"
server_key_iv: "1e045c1c41e712af9b4ac67f06771cdb"
server_key_version: "encryptionkey@0"
I20260927 10:49:45.181604 8353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:49:45.184020 8368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:45.185237 8353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:45.185374 8353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "2fe85a810a1b4c718f64b1d60debc2f8"
format_stamp: "Formatted at 2026-09-27 10:49:45 on dist-test-slave-c60x"
server_key: "b8ec08f6e3d9b8cc92bd63f424f27af0"
server_key_iv: "1e045c1c41e712af9b4ac67f06771cdb"
server_key_version: "encryptionkey@0"
I20260927 10:49:45.185530 8353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:45.200232 8353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:45.203033 8353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:45.203265 8353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:45.204779 8353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:45.204849 8353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:45.204919 8353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:45.204953 8353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:49:45 dist-test-slave-c60x krb5kdc[7956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506185, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:45.215893 8353 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:33069
I20260927 10:49:45.216463 8483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:33069 every 8 connection(s)
I20260927 10:49:45.216975 8353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:49:45.219753 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.205503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:45151 (local address 127.6.31.190:33397)
0927 10:49:45.205644 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.205648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.206814 (+ 1166us) server_negotiation.cc:407] Connection header received
0927 10:49:45.208065 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.208069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.208133 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.208273 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:45.209801 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.210460 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.212087 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.212348 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.212476 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.215276 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:45.215280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:45.215283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:45.215320 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:45.217244 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.217970 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.217977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.217979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.218078 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.218535 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.218538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.218539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.218752 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:45.218831 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.219460 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.219564 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:49:45.220572 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.206063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33397 (local address 127.6.31.131:45151)
0927 10:49:45.206573 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.206618 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.207813 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.208451 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.208457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.208892 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:45.209617 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.209638 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.210642 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.210648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.211941 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.211955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.212535 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.212540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.212725 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.213380 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:45.213398 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:45.215000 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:45.217376 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:45.217384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:45.217402 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:45.217814 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:45.218202 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:45.218206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:45.218211 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:45.218404 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:45.218860 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:45.218867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:45.219222 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.219387 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.219659 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":133,"threads_started":1}
I20260927 10:49:45.221859 8484 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33397
I20260927 10:49:45.222004 8484 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:45.222599 8484 heartbeater.cc:511] Master 127.6.31.190:33397 requested a full tablet report, sending...
I20260927 10:49:45.223819 8000 ts_manager.cc:194] Registered new tserver with Master: 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069)
I20260927 10:49:45.224598 8000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:45151
I20260927 10:49:45.226397 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 8353
I20260927 10:49:45.226527 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:49:45.226817 6270 external_mini_cluster.cc:1428] Setting key 92c622dcc9f392e6b89749de0ed850da
I20260927 10:49:45.228889 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:49:45.231619 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.224458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:43057 (local address 127.6.31.190:33397)
0927 10:49:45.224577 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.224582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.224688 (+ 106us) server_negotiation.cc:407] Connection header received
0927 10:49:45.224831 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.224834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.224876 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.224973 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:45.225491 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.225968 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.226975 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.227281 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.227395 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.228416 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:45.228420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:45.228422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:45.228445 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:45.229945 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.230373 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.230378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.230380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.230504 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.230786 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.230791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.230794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.230973 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:45.231083 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.231374 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.231483 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:49:45.231630 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.224320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33397 (local address 127.6.31.131:43057)
0927 10:49:45.224593 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.224612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.224735 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.224987 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.224989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.225251 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:45.225369 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.225376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.226093 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.226097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.226838 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.226847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.227398 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.227401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.227491 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.227975 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:45.227986 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:45.228288 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:45.230072 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:45.230076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:45.230078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:45.230271 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:45.230601 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:45.230605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:45.230607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:45.230695 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:45.231070 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:45.231073 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:45.231197 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.231309 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.231451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
Sep 27 10:49:45 dist-test-slave-c60x krb5kdc[7956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:49:45 dist-test-slave-c60x krb5kdc[7956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:45.254540 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.246445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.6.31.190:33397)
0927 10:49:45.246575 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.246579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.246612 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:49:45.246658 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.246664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.246708 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.246781 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:45.247579 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.248083 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.248945 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.249141 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.249212 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.251595 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:45.251598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:45.251600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:45.251622 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:45.253055 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.253423 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.253425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.253427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.253494 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.253718 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.253720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.253721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.253888 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:45.253988 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.254222 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.254316 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:49:45.257598 8000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:49:45.260701 8000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:49:45.275018 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.269162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42713 (local address 127.0.0.1:34456)
0927 10:49:45.269994 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.270010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.270328 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.270842 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.270851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.270879 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:45.271172 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.271180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.272257 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.272264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.273527 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.273536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.274426 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.274430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.274532 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.274573 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.274714 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.274856 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":68,"thread_start_us":95,"threads_started":1}
I20260927 10:49:45.275039 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.269856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.6.31.129:42713)
0927 10:49:45.270231 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.270249 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.270277 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:49:45.270475 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.270482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.270678 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.270855 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:45.271285 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.272145 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.273647 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.274299 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.274408 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.274567 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.274771 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.274865 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":96,"threads_started":1}
I20260927 10:49:45.275283 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.269083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:33069 (local address 127.0.0.1:58258)
0927 10:49:45.269621 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.269671 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.269798 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.270897 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.270901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.270918 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:45.271153 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.271161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.272644 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.272647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.273921 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.273931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.274883 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.274886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.274981 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.275003 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.275050 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.275149 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":28,"thread_start_us":82,"threads_started":1}
I20260927 10:49:45.275928 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.269742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:46539 (local address 127.0.0.1:33274)
0927 10:49:45.270225 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.270238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.270323 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.271014 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.271016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.271031 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:45.271227 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.271233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.272644 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.272647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.273986 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.273993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.275506 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.275509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.275612 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.275629 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.275724 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.275794 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":91,"thread_start_us":154,"threads_started":1}
I20260927 10:49:45.276059 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.270184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.6.31.130:46539)
0927 10:49:45.270549 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.270557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.270575 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:49:45.270632 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.270636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.270793 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.270941 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:45.271348 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.272511 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.274103 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.274954 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.275069 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.275470 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.275754 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.275840 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":85,"threads_started":1}
I20260927 10:49:45.276669 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.269203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.6.31.131:33069)
0927 10:49:45.270477 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.270488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.270506 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:49:45.270566 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.270570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.270705 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.271295 (+ 590us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:45.271427 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.272517 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.274077 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.274761 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.276247 (+ 1486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.276419 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.276464 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.276542 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":101,"threads_started":1}
I20260927 10:49:45.278255 8138 tablet_service.cc:1467] Processing CreateTablet for tablet b493a0e4b4b544c59ea7b089693cd350 (DEFAULT_TABLE table=test-table [id=905dcec962724422b680a5385d31262b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:45.278514 8276 tablet_service.cc:1467] Processing CreateTablet for tablet b493a0e4b4b544c59ea7b089693cd350 (DEFAULT_TABLE table=test-table [id=905dcec962724422b680a5385d31262b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:45.279343 8138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b493a0e4b4b544c59ea7b089693cd350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:45.278968 8418 tablet_service.cc:1467] Processing CreateTablet for tablet b493a0e4b4b544c59ea7b089693cd350 (DEFAULT_TABLE table=test-table [id=905dcec962724422b680a5385d31262b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:45.279510 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b493a0e4b4b544c59ea7b089693cd350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:45.279918 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b493a0e4b4b544c59ea7b089693cd350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:45.285827 8507 tablet_bootstrap.cc:492] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: Bootstrap starting.
I20260927 10:49:45.287361 8508 tablet_bootstrap.cc:492] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: Bootstrap starting.
I20260927 10:49:45.287361 8509 tablet_bootstrap.cc:492] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: Bootstrap starting.
I20260927 10:49:45.288543 8507 tablet_bootstrap.cc:654] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:45.289322 8509 tablet_bootstrap.cc:654] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:45.289322 8508 tablet_bootstrap.cc:654] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:45.289407 8507 log.cc:824] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:45.290122 8509 log.cc:824] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:45.290122 8508 log.cc:824] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:45.291657 8507 tablet_bootstrap.cc:492] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: No bootstrap required, opened a new log
I20260927 10:49:45.291942 8507 ts_tablet_manager.cc:1397] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:49:45.292062 8508 tablet_bootstrap.cc:492] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: No bootstrap required, opened a new log
I20260927 10:49:45.292239 8508 ts_tablet_manager.cc:1397] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 10:49:45.292435 8509 tablet_bootstrap.cc:492] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: No bootstrap required, opened a new log
I20260927 10:49:45.292598 8509 ts_tablet_manager.cc:1397] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 10:49:45.294822 8507 raft_consensus.cc:348] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } }
I20260927 10:49:45.295058 8507 raft_consensus.cc:374] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:45.295131 8507 raft_consensus.cc:729] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5445a4527d64988ab038632451dff78, State: Initialized, Role: FOLLOWER
I20260927 10:49:45.295563 8507 consensus_queue.cc:260] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } }
I20260927 10:49:45.296166 8204 heartbeater.cc:503] Master 127.6.31.190:33397 was elected leader, sending a full tablet report...
I20260927 10:49:45.296034 8508 raft_consensus.cc:348] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } }
I20260927 10:49:45.296270 8508 raft_consensus.cc:374] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:45.296339 8508 raft_consensus.cc:729] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2300a0bbf8a4cffa903cf48f8083400, State: Initialized, Role: FOLLOWER
I20260927 10:49:45.296528 8509 raft_consensus.cc:348] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } }
I20260927 10:49:45.296748 8509 raft_consensus.cc:374] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:45.296798 8509 raft_consensus.cc:729] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe85a810a1b4c718f64b1d60debc2f8, State: Initialized, Role: FOLLOWER
I20260927 10:49:45.296730 8508 consensus_queue.cc:260] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } }
I20260927 10:49:45.297175 8509 consensus_queue.cc:260] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } }
I20260927 10:49:45.297662 8508 ts_tablet_manager.cc:1428] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:49:45.297663 8507 ts_tablet_manager.cc:1428] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260927 10:49:45.297983 8344 heartbeater.cc:503] Master 127.6.31.190:33397 was elected leader, sending a full tablet report...
I20260927 10:49:45.298048 8509 ts_tablet_manager.cc:1428] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260927 10:49:45.298374 8484 heartbeater.cc:503] Master 127.6.31.190:33397 was elected leader, sending a full tablet report...
W20260927 10:49:45.467980 8485 tablet.cc:2369] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:49:45.480515 8514 raft_consensus.cc:482] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:45.480847 8514 raft_consensus.cc:504] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } }
I20260927 10:49:45.482213 8514 leader_election.cc:288] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069), b5445a4527d64988ab038632451dff78 (127.6.31.129:42713)
I20260927 10:49:45.486780 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.482500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:33069 (local address 127.6.31.130:34989)
0927 10:49:45.482723 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.482738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.482848 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.483172 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.483175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.483195 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:45.483343 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.483350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.484243 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.484246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.485485 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.485496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.486339 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.486342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.486463 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.486481 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.486570 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.486646 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":3,"spinlock_wait_cycles":1280}
I20260927 10:49:45.486779 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.482717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:34989 (local address 127.6.31.131:33069)
0927 10:49:45.482834 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.482838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.482855 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:49:45.482974 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.482977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.483032 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.483161 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:45.483485 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.484123 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.485632 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.486205 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.486339 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.486397 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.486585 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.486658 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:49:45.487468 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.482565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:49129 (local address 127.6.31.129:42713)
0927 10:49:45.482726 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.482733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.483174 (+ 441us) server_negotiation.cc:407] Connection header received
0927 10:49:45.483367 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.483371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.483420 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.483494 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:45.483878 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.484489 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.486079 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.486650 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.486787 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.486835 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.487281 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.487356 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 10:49:45.487471 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.482500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42713 (local address 127.6.31.130:49129)
0927 10:49:45.483056 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.483067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.483250 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.483595 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.483598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.483614 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:45.483746 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.483752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.484609 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.484615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.485946 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.485956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.487040 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.487044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.487165 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.487188 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.487265 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.487330 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":254,"spinlock_wait_cycles":2560,"thread_start_us":124,"threads_started":1}
I20260927 10:49:45.487509 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b493a0e4b4b544c59ea7b089693cd350" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" is_pre_election: true
I20260927 10:49:45.487823 8438 raft_consensus.cc:2454] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 0.
I20260927 10:49:45.487998 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b493a0e4b4b544c59ea7b089693cd350" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5445a4527d64988ab038632451dff78" is_pre_election: true
I20260927 10:49:45.488305 8158 raft_consensus.cc:2454] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 0.
I20260927 10:49:45.488373 8231 leader_election.cc:302] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe85a810a1b4c718f64b1d60debc2f8, a2300a0bbf8a4cffa903cf48f8083400; no voters:
I20260927 10:49:45.488684 8514 raft_consensus.cc:2781] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:49:45.488785 8514 raft_consensus.cc:482] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:49:45.488837 8514 raft_consensus.cc:3037] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:45.489877 8514 raft_consensus.cc:504] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } }
I20260927 10:49:45.490280 8514 leader_election.cc:288] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 election: Requested vote from peers 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069), b5445a4527d64988ab038632451dff78 (127.6.31.129:42713)
I20260927 10:49:45.490607 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b493a0e4b4b544c59ea7b089693cd350" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe85a810a1b4c718f64b1d60debc2f8"
I20260927 10:49:45.490650 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b493a0e4b4b544c59ea7b089693cd350" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5445a4527d64988ab038632451dff78"
I20260927 10:49:45.490754 8438 raft_consensus.cc:3037] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:45.490780 8158 raft_consensus.cc:3037] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:45.492035 8438 raft_consensus.cc:2454] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 1.
I20260927 10:49:45.492048 8158 raft_consensus.cc:2454] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 1.
I20260927 10:49:45.492349 8230 leader_election.cc:302] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2300a0bbf8a4cffa903cf48f8083400, b5445a4527d64988ab038632451dff78; no voters:
I20260927 10:49:45.492511 8514 raft_consensus.cc:2781] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:45.492767 8514 raft_consensus.cc:686] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 LEADER]: Becoming Leader. State: Replica: a2300a0bbf8a4cffa903cf48f8083400, State: Running, Role: LEADER
I20260927 10:49:45.493119 8514 consensus_queue.cc:237] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } }
W20260927 10:49:45.493400 8345 tablet.cc:2369] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:49:45.496546 7999 catalog_manager.cc:5495] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "a2300a0bbf8a4cffa903cf48f8083400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } health_report { overall_health: HEALTHY } } }
W20260927 10:49:45.526693 8205 tablet.cc:2369] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:49:45 dist-test-slave-c60x krb5kdc[7956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506184, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:45.537720 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.528863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33397 (local address 127.0.0.1:47966)
0927 10:49:45.529146 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.529158 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.529261 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.529522 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.529525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.529891 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:45.529982 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.529987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.530717 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.530719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.531370 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.531378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.531867 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.531870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.531944 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.532498 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:45.532510 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:45.534181 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:45.536086 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:45.536095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:45.536100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:45.536345 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:45.536594 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:45.536596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:45.536597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:45.536672 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:45.537254 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:45.537259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:45.537364 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.537463 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.537579 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":110,"threads_started":1}
I20260927 10:49:45.537755 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.528906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.6.31.190:33397)
0927 10:49:45.529041 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.529045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.529260 (+ 215us) server_negotiation.cc:407] Connection header received
0927 10:49:45.529330 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.529332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.529372 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.529456 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:45.530089 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.530623 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.531504 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.531726 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.531815 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.534373 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:45.534376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:45.534378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:45.534399 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:45.535981 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.536437 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.536440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.536441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.536512 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:45.536953 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:45.536956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:45.536957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:45.537152 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:45.537240 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.537551 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.537647 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:49:45.542095 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:47966:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:49:45.542762 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:49:45.548080 8276 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd0c75dcd044970a6fbb611160607d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92c055cee5a84b9497aa0b869dc4076c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:49:45.548082 8138 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd0c75dcd044970a6fbb611160607d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92c055cee5a84b9497aa0b869dc4076c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:49:45.548250 8418 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd0c75dcd044970a6fbb611160607d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92c055cee5a84b9497aa0b869dc4076c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:49:45.548859 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd0c75dcd044970a6fbb611160607d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:45.548892 8138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd0c75dcd044970a6fbb611160607d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:45.548918 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd0c75dcd044970a6fbb611160607d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:45.551621 8509 tablet_bootstrap.cc:492] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: Bootstrap starting.
I20260927 10:49:45.551666 8507 tablet_bootstrap.cc:492] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: Bootstrap starting.
I20260927 10:49:45.551666 8508 tablet_bootstrap.cc:492] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: Bootstrap starting.
I20260927 10:49:45.552493 8509 tablet_bootstrap.cc:654] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:45.552618 8507 tablet_bootstrap.cc:654] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:45.552618 8508 tablet_bootstrap.cc:654] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:45.554106 8508 tablet_bootstrap.cc:492] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: No bootstrap required, opened a new log
I20260927 10:49:45.554199 8508 ts_tablet_manager.cc:1397] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 10:49:45.554382 8509 tablet_bootstrap.cc:492] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: No bootstrap required, opened a new log
I20260927 10:49:45.554479 8509 ts_tablet_manager.cc:1397] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 10:49:45.554576 8507 tablet_bootstrap.cc:492] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: No bootstrap required, opened a new log
I20260927 10:49:45.554663 8507 ts_tablet_manager.cc:1397] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 10:49:45.554674 8508 raft_consensus.cc:348] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.554800 8508 raft_consensus.cc:374] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:45.554844 8508 raft_consensus.cc:729] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2300a0bbf8a4cffa903cf48f8083400, State: Initialized, Role: FOLLOWER
I20260927 10:49:45.554937 8509 raft_consensus.cc:348] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.554970 8508 consensus_queue.cc:260] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.555063 8509 raft_consensus.cc:374] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:45.555137 8509 raft_consensus.cc:729] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe85a810a1b4c718f64b1d60debc2f8, State: Initialized, Role: FOLLOWER
I20260927 10:49:45.555187 8507 raft_consensus.cc:348] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.555312 8507 raft_consensus.cc:374] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:45.555279 8509 consensus_queue.cc:260] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.555258 8514 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.555356 8507 raft_consensus.cc:729] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5445a4527d64988ab038632451dff78, State: Initialized, Role: FOLLOWER
I20260927 10:49:45.555416 8514 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:45.555472 8507 consensus_queue.cc:260] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.555581 8509 ts_tablet_manager.cc:1428] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:45.555735 8507 ts_tablet_manager.cc:1428] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:45.555951 8513 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.555984 8515 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.556087 8513 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:45.556137 8515 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:45.556501 8508 ts_tablet_manager.cc:1428] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 10:49:45.654419 8515 raft_consensus.cc:482] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:45.654595 8515 raft_consensus.cc:504] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.655865 8515 leader_election.cc:288] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5445a4527d64988ab038632451dff78 (127.6.31.129:42713), a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130:46539)
I20260927 10:49:45.661130 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.656156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:38207 (local address 127.6.31.129:42713)
0927 10:49:45.656280 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.656284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.656423 (+ 139us) server_negotiation.cc:407] Connection header received
0927 10:49:45.656497 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.656500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.656541 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.656654 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:45.656969 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.657751 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.659772 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.660405 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.660544 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.660598 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.660925 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.661006 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:49:45.661130 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.656231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:44355 (local address 127.6.31.130:46539)
0927 10:49:45.656367 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.656372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.656595 (+ 223us) server_negotiation.cc:407] Connection header received
0927 10:49:45.656677 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.656680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.656730 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.656805 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:45.657087 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.657815 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.659772 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.660410 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.660544 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.660598 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.660936 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.661006 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:49:45.661113 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.656061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42713 (local address 127.6.31.131:38207)
0927 10:49:45.656252 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.656268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.656394 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.656680 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.656683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.656703 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:45.656832 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.656838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.657921 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.657924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.659601 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.659614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.660550 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.660554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.660720 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.660758 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.660842 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.660946 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":219}
I20260927 10:49:45.661128 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.656140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:46539 (local address 127.6.31.131:44355)
0927 10:49:45.656494 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.656511 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.656584 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.656838 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.656841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.656855 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:45.656985 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.656990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.657921 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.657924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.659601 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.659614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.660555 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.660558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.660723 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.660837 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.660903 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.660959 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":270,"thread_start_us":120,"threads_started":1}
I20260927 10:49:45.661742 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd0c75dcd044970a6fbb611160607d2" candidate_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5445a4527d64988ab038632451dff78" is_pre_election: true
I20260927 10:49:45.661824 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd0c75dcd044970a6fbb611160607d2" candidate_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2300a0bbf8a4cffa903cf48f8083400" is_pre_election: true
I20260927 10:49:45.661890 8158 raft_consensus.cc:2454] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe85a810a1b4c718f64b1d60debc2f8 in term 0.
I20260927 10:49:45.662101 8298 raft_consensus.cc:2454] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe85a810a1b4c718f64b1d60debc2f8 in term 0.
I20260927 10:49:45.662329 8370 leader_election.cc:302] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe85a810a1b4c718f64b1d60debc2f8, b5445a4527d64988ab038632451dff78; no voters:
I20260927 10:49:45.662637 8515 raft_consensus.cc:2781] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:49:45.662696 8515 raft_consensus.cc:482] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:49:45.662724 8515 raft_consensus.cc:3037] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:45.663720 8515 raft_consensus.cc:504] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.664103 8515 leader_election.cc:288] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [CANDIDATE]: Term 1 election: Requested vote from peers b5445a4527d64988ab038632451dff78 (127.6.31.129:42713), a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130:46539)
I20260927 10:49:45.664407 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd0c75dcd044970a6fbb611160607d2" candidate_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5445a4527d64988ab038632451dff78"
I20260927 10:49:45.664438 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd0c75dcd044970a6fbb611160607d2" candidate_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2300a0bbf8a4cffa903cf48f8083400"
I20260927 10:49:45.664527 8298 raft_consensus.cc:3037] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:45.664525 8158 raft_consensus.cc:3037] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:45.665606 8158 raft_consensus.cc:2454] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe85a810a1b4c718f64b1d60debc2f8 in term 1.
I20260927 10:49:45.665606 8298 raft_consensus.cc:2454] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe85a810a1b4c718f64b1d60debc2f8 in term 1.
I20260927 10:49:45.665863 8370 leader_election.cc:302] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2fe85a810a1b4c718f64b1d60debc2f8, a2300a0bbf8a4cffa903cf48f8083400, b5445a4527d64988ab038632451dff78; no voters:
I20260927 10:49:45.666040 8515 raft_consensus.cc:2781] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:45.666316 8515 raft_consensus.cc:686] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 LEADER]: Becoming Leader. State: Replica: 2fe85a810a1b4c718f64b1d60debc2f8, State: Running, Role: LEADER
I20260927 10:49:45.666663 8515 consensus_queue.cc:237] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.669202 8529 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: TxnStatusTablet state changed. Reason: New leader 2fe85a810a1b4c718f64b1d60debc2f8. Latest consensus state: current_term: 1 leader_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.669315 8529 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:49:45.669580 8531 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:49:45.669783 8531 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:49:45.670377 7999 catalog_manager.cc:5495] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131). New cstate: current_term: 1 leader_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } health_report { overall_health: HEALTHY } } }
I20260927 10:49:45.712013 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.708009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:33069 (local address 127.0.0.1:58260)
0927 10:49:45.708175 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.708192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.708310 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.708652 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.708655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.708662 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:49:45.708769 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.708776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.709503 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.709508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.710182 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.710195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.710816 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.710820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.710886 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.710929 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:49:45.711750 (+ 821us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:49:45.711762 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.711828 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.711896 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:49:45.712039 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.708159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.6.31.131:33069)
0927 10:49:45.708281 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.708286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.708301 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:49:45.708464 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.708474 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.708528 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.708628 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:45.708881 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.709394 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.710315 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.710673 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.710783 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.711130 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:45.711654 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:45.711765 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.711851 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.711906 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:49:45.717300 8158 raft_consensus.cc:1261] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Refusing update from remote peer 2fe85a810a1b4c718f64b1d60debc2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:45.717338 8298 raft_consensus.cc:1261] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Refusing update from remote peer 2fe85a810a1b4c718f64b1d60debc2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:45.717979 8529 consensus_queue.cc:1035] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:45.718146 8515 consensus_queue.cc:1035] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:45.721403 8514 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: TxnStatusTablet state changed. Reason: New leader 2fe85a810a1b4c718f64b1d60debc2f8. Latest consensus state: current_term: 1 leader_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.721539 8514 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:45.721668 8513 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: TxnStatusTablet state changed. Reason: New leader 2fe85a810a1b4c718f64b1d60debc2f8. Latest consensus state: current_term: 1 leader_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.721820 8513 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:45.725210 8529 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.725330 8529 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:49:45.725586 8513 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.725689 8513 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P b5445a4527d64988ab038632451dff78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:45.726965 8536 mvcc.cc:205] Tried to move back new op lower bound from 7333913336691867648 to 7333913336493436928. Current Snapshot: MvccSnapshot[applied={T|T < 7333913336691867648}]
I20260927 10:49:45.727013 8515 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.727133 8515 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P 2fe85a810a1b4c718f64b1d60debc2f8: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:49:45.728164 8514 tablet_replica.cc:439] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } } }
I20260927 10:49:45.728283 8514 tablet_replica.cc:442] T 7bd0c75dcd044970a6fbb611160607d2 P a2300a0bbf8a4cffa903cf48f8083400: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:49:45.740306 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.736072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.6.31.130:46539)
0927 10:49:45.736423 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.736428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.736440 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:49:45.736507 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.736510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.736566 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.736665 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:45.736964 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.737752 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.738608 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.738945 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.739046 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.739386 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:45.739907 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:45.740015 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.740114 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.740170 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260927 10:49:45.751364 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.747901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:51281 (local address 127.6.31.131:33069)
0927 10:49:45.748019 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.748025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.748103 (+ 78us) server_negotiation.cc:407] Connection header received
0927 10:49:45.748236 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.748239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.748286 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.748345 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:45.748602 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.749122 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.750331 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.750532 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.750607 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.750953 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:45.751018 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:45.751072 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.751215 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.751256 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:49:45.751446 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.747713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:33069 (local address 127.6.31.130:51281)
0927 10:49:45.748017 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.748033 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.748133 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.748373 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.748376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.748385 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:49:45.748505 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.748511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.749219 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.749222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.750219 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.750229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.750633 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.750637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.750732 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.750805 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:49:45.751120 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:49:45.751126 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.751193 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.751269 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
I20260927 10:49:45.758149 8158 raft_consensus.cc:1261] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Refusing update from remote peer a2300a0bbf8a4cffa903cf48f8083400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:45.758216 8438 raft_consensus.cc:1261] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Refusing update from remote peer a2300a0bbf8a4cffa903cf48f8083400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:45.758653 8517 consensus_queue.cc:1035] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:45.758872 8514 consensus_queue.cc:1035] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:45.761973 8535 mvcc.cc:205] Tried to move back new op lower bound from 7333913336860823552 to 7333913335782916096. Current Snapshot: MvccSnapshot[applied={T|T < 7333913336860823552}]
I20260927 10:49:45.764967 8534 mvcc.cc:205] Tried to move back new op lower bound from 7333913336860823552 to 7333913335782916096. Current Snapshot: MvccSnapshot[applied={T|T < 7333913336860823552}]
I20260927 10:49:45.791939 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.782428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:58489 (local address 127.6.31.130:46539)
0927 10:49:45.782542 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:45.782547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:45.787524 (+ 4977us) server_negotiation.cc:407] Connection header received
0927 10:49:45.787720 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:45.787724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:45.787778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:45.787825 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:45.788346 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.789202 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.790582 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.790919 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.791022 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.791478 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:45.791562 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:45.791603 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:45.791745 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 10:49:45.791802 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:49:45.792397 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:45.782332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:46539 (local address 127.6.31.131:58489)
0927 10:49:45.787307 (+ 4975us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:45.787461 (+ 154us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:45.787606 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:45.788027 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:45.788030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:45.788038 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:49:45.788167 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.788173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.789324 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.789329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.790419 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:45.790430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:45.791145 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:45.791149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:45.791255 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:45.791311 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:49:45.791657 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:49:45.791662 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:49:45.792153 (+ 491us) client_negotiation.cc:240] Negotiation successful
0927 10:49:45.792235 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4839,"thread_start_us":88,"threads_started":1}
I20260927 10:49:45.820968 8001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:45.821167 8001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:45.824059 8001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f: Sending DeleteTablet for 3 replicas of tablet b493a0e4b4b544c59ea7b089693cd350
I20260927 10:49:45.824769 8418 tablet_service.cc:1514] Processing DeleteTablet for tablet b493a0e4b4b544c59ea7b089693cd350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:45 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:58258
I20260927 10:49:45.825011 8138 tablet_service.cc:1514] Processing DeleteTablet for tablet b493a0e4b4b544c59ea7b089693cd350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:45 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:34456
I20260927 10:49:45.825444 8555 tablet_replica.cc:330] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: stopping tablet replica
I20260927 10:49:45.825693 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet b493a0e4b4b544c59ea7b089693cd350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:45 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:33274
I20260927 10:49:45.825796 8555 raft_consensus.cc:2229] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:45.826092 8555 raft_consensus.cc:2258] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:45.826908 8001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:49:45.827375 8001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:49:45.827751 8555 ts_tablet_manager.cc:1905] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:45.830863 8555 ts_tablet_manager.cc:1918] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:49:45.831035 8555 log.cc:1196] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/b493a0e4b4b544c59ea7b089693cd350
I20260927 10:49:45.831444 8555 ts_tablet_manager.cc:1939] T b493a0e4b4b544c59ea7b089693cd350 P b5445a4527d64988ab038632451dff78: Deleting consensus metadata
I20260927 10:49:45.832638 8418 tablet_service.cc:1467] Processing CreateTablet for tablet f04eb2451e7b458abad5621a7c15e661 (DEFAULT_TABLE table=test-table [id=298208d7a9b24a58bb3c3d4b55d78126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:45.832680 7986 catalog_manager.cc:4841] TS b5445a4527d64988ab038632451dff78 (127.6.31.129:42713): tablet b493a0e4b4b544c59ea7b089693cd350 (table test-table [id=905dcec962724422b680a5385d31262b]) successfully deleted
I20260927 10:49:45.832921 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04eb2451e7b458abad5621a7c15e661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:45.835348 8509 tablet_bootstrap.cc:492] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: Bootstrap starting.
I20260927 10:49:45.835592 8138 tablet_service.cc:1467] Processing CreateTablet for tablet f04eb2451e7b458abad5621a7c15e661 (DEFAULT_TABLE table=test-table [id=298208d7a9b24a58bb3c3d4b55d78126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:45.835852 8138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04eb2451e7b458abad5621a7c15e661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:45.836162 8509 tablet_bootstrap.cc:654] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:45.836941 8276 tablet_service.cc:1467] Processing CreateTablet for tablet f04eb2451e7b458abad5621a7c15e661 (DEFAULT_TABLE table=test-table [id=298208d7a9b24a58bb3c3d4b55d78126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:45.837188 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04eb2451e7b458abad5621a7c15e661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:45.838495 8556 tablet_replica.cc:330] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: stopping tablet replica
I20260927 10:49:45.838845 8556 raft_consensus.cc:2229] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:49:45.839210 8507 tablet_bootstrap.cc:492] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: Bootstrap starting.
I20260927 10:49:45.839318 8556 raft_consensus.cc:2258] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:45.840081 8507 tablet_bootstrap.cc:654] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:45.840651 8556 ts_tablet_manager.cc:1905] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:45.840875 8554 tablet_replica.cc:330] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: stopping tablet replica
I20260927 10:49:45.841166 8554 raft_consensus.cc:2229] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:45.841341 8554 raft_consensus.cc:2258] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:45.842077 8508 tablet_bootstrap.cc:492] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: Bootstrap starting.
I20260927 10:49:45.842938 8507 tablet_bootstrap.cc:492] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: No bootstrap required, opened a new log
I20260927 10:49:45.843036 8507 ts_tablet_manager.cc:1397] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 10:49:45.843482 8508 tablet_bootstrap.cc:654] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:45.843467 8507 raft_consensus.cc:348] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.843573 8507 raft_consensus.cc:374] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:45.843603 8507 raft_consensus.cc:729] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5445a4527d64988ab038632451dff78, State: Initialized, Role: FOLLOWER
I20260927 10:49:45.843716 8507 consensus_queue.cc:260] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.843983 8507 ts_tablet_manager.cc:1428] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:45.845770 8554 ts_tablet_manager.cc:1905] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:45.847440 8509 tablet_bootstrap.cc:492] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: No bootstrap required, opened a new log
I20260927 10:49:45.847528 8509 ts_tablet_manager.cc:1397] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260927 10:49:45.847929 8509 raft_consensus.cc:348] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.848043 8509 raft_consensus.cc:374] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:45.848079 8509 raft_consensus.cc:729] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe85a810a1b4c718f64b1d60debc2f8, State: Initialized, Role: FOLLOWER
I20260927 10:49:45.848196 8509 consensus_queue.cc:260] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.848443 8509 ts_tablet_manager.cc:1428] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:45.850224 8554 ts_tablet_manager.cc:1918] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:49:45.850318 8554 log.cc:1196] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/b493a0e4b4b544c59ea7b089693cd350
I20260927 10:49:45.850641 8554 ts_tablet_manager.cc:1939] T b493a0e4b4b544c59ea7b089693cd350 P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting consensus metadata
I20260927 10:49:45.851735 8556 ts_tablet_manager.cc:1918] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:49:45.851794 7987 catalog_manager.cc:4841] TS 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069): tablet b493a0e4b4b544c59ea7b089693cd350 (table test-table [id=905dcec962724422b680a5385d31262b]) successfully deleted
I20260927 10:49:45.851945 8556 log.cc:1196] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/b493a0e4b4b544c59ea7b089693cd350
I20260927 10:49:45.852267 8556 ts_tablet_manager.cc:1939] T b493a0e4b4b544c59ea7b089693cd350 P a2300a0bbf8a4cffa903cf48f8083400: Deleting consensus metadata
I20260927 10:49:45.853235 7987 catalog_manager.cc:4841] TS a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130:46539): tablet b493a0e4b4b544c59ea7b089693cd350 (table test-table [id=905dcec962724422b680a5385d31262b]) successfully deleted
I20260927 10:49:45.853734 8508 tablet_bootstrap.cc:492] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: No bootstrap required, opened a new log
I20260927 10:49:45.853807 8508 ts_tablet_manager.cc:1397] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260927 10:49:45.854203 8508 raft_consensus.cc:348] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.854297 8508 raft_consensus.cc:374] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:45.854319 8508 raft_consensus.cc:729] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2300a0bbf8a4cffa903cf48f8083400, State: Initialized, Role: FOLLOWER
I20260927 10:49:45.854478 8508 consensus_queue.cc:260] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:45.854741 8508 ts_tablet_manager.cc:1428] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:46.039213 8513 raft_consensus.cc:482] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:46.039352 8513 raft_consensus.cc:504] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:46.040393 8513 leader_election.cc:288] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130:46539), 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069)
I20260927 10:49:46.044598 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:46.041049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:53663 (local address 127.6.31.130:46539)
0927 10:49:46.041198 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:46.041203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:46.041219 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:49:46.041299 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:46.041302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:46.041354 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:46.041454 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:46.041742 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.042281 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.043584 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.044000 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.044146 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:46.044214 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:46.044412 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 10:49:46.044478 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:49:46.044626 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:46.040673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:46539 (local address 127.6.31.129:53663)
0927 10:49:46.041004 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:46.041021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:46.041127 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:46.041479 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:46.041482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:46.041501 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:46.041625 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:46.041630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.042391 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.042395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:46.043454 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:46.043463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.044144 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.044147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:46.044287 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:46.044305 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:49:46.044393 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:49:46.044464 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"spinlock_wait_cycles":27904,"thread_start_us":149,"threads_started":1}
I20260927 10:49:46.044814 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:46.041322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:48935 (local address 127.6.31.131:33069)
0927 10:49:46.041486 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:46.041493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:46.041510 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:49:46.041682 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:46.041685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:46.041731 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:46.041815 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:49:46.042070 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.042633 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.043793 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.044269 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.044425 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:46.044478 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:46.044639 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:49:46.044703 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 10:49:46.044838 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:46.041143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:33069 (local address 127.6.31.129:48935)
0927 10:49:46.041446 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:46.041459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:46.041574 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:46.041825 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:46.041828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:46.041842 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:49:46.041967 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:46.041972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.042760 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.042764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:46.043677 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:46.043686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.044398 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.044402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:46.044509 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:46.044529 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:49:46.044624 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:49:46.044691 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260927 10:49:46.045240 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04eb2451e7b458abad5621a7c15e661" candidate_uuid: "b5445a4527d64988ab038632451dff78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2300a0bbf8a4cffa903cf48f8083400" is_pre_election: true
I20260927 10:49:46.045411 8298 raft_consensus.cc:2454] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5445a4527d64988ab038632451dff78 in term 0.
I20260927 10:49:46.045674 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04eb2451e7b458abad5621a7c15e661" candidate_uuid: "b5445a4527d64988ab038632451dff78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" is_pre_election: true
I20260927 10:49:46.045833 8438 raft_consensus.cc:2454] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5445a4527d64988ab038632451dff78 in term 0.
I20260927 10:49:46.045796 8091 leader_election.cc:302] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2300a0bbf8a4cffa903cf48f8083400, b5445a4527d64988ab038632451dff78; no voters:
I20260927 10:49:46.046067 8513 raft_consensus.cc:2781] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:49:46.046130 8513 raft_consensus.cc:482] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:49:46.046168 8513 raft_consensus.cc:3037] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:46.047245 8513 raft_consensus.cc:504] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:46.047570 8513 leader_election.cc:288] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [CANDIDATE]: Term 1 election: Requested vote from peers a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130:46539), 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069)
I20260927 10:49:46.047873 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04eb2451e7b458abad5621a7c15e661" candidate_uuid: "b5445a4527d64988ab038632451dff78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2300a0bbf8a4cffa903cf48f8083400"
I20260927 10:49:46.047914 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04eb2451e7b458abad5621a7c15e661" candidate_uuid: "b5445a4527d64988ab038632451dff78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe85a810a1b4c718f64b1d60debc2f8"
I20260927 10:49:46.047991 8298 raft_consensus.cc:3037] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:46.048010 8438 raft_consensus.cc:3037] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:46.049093 8298 raft_consensus.cc:2454] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5445a4527d64988ab038632451dff78 in term 1.
I20260927 10:49:46.049093 8438 raft_consensus.cc:2454] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5445a4527d64988ab038632451dff78 in term 1.
I20260927 10:49:46.049374 8091 leader_election.cc:302] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2300a0bbf8a4cffa903cf48f8083400, b5445a4527d64988ab038632451dff78; no voters:
I20260927 10:49:46.049539 8513 raft_consensus.cc:2781] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:46.049731 8513 raft_consensus.cc:686] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 1 LEADER]: Becoming Leader. State: Replica: b5445a4527d64988ab038632451dff78, State: Running, Role: LEADER
I20260927 10:49:46.049989 8513 consensus_queue.cc:237] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:46.052172 8001 catalog_manager.cc:5495] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5445a4527d64988ab038632451dff78 (127.6.31.129). New cstate: current_term: 1 leader_uuid: "b5445a4527d64988ab038632451dff78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } health_report { overall_health: UNKNOWN } } }
I20260927 10:49:46.093150 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:46.089746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.6.31.129:42713)
0927 10:49:46.089855 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:46.089861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:46.089940 (+ 79us) server_negotiation.cc:407] Connection header received
0927 10:49:46.090027 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:46.090030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:46.090088 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:46.090174 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:46.090443 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.090931 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.091788 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.092008 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.092086 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:46.092288 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:46.092744 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:46.092828 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:46.092925 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 10:49:46.092971 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:49:46.103369 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:46.100132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:47411 (local address 127.6.31.131:33069)
0927 10:49:46.100387 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:46.100390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:46.100515 (+ 125us) server_negotiation.cc:407] Connection header received
0927 10:49:46.100623 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:46.100625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:46.100666 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:46.100727 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:46.100915 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.101424 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.102195 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.102407 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.102487 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:46.102882 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:46.102963 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:46.103029 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:46.103219 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 10:49:46.103266 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260927 10:49:46.103394 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:46.100056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:33069 (local address 127.6.31.129:47411)
0927 10:49:46.100455 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:46.100468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:46.100554 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:46.100750 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:46.100752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:46.100758 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:49:46.100851 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:46.100855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.101513 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.101516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:46.102093 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:46.102100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.102602 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.102605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:46.102681 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:46.102726 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:49:46.103054 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:49:46.103060 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:49:46.103189 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 10:49:46.103258 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":101,"threads_started":1}
I20260927 10:49:46.109988 8438 raft_consensus.cc:1261] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Refusing update from remote peer b5445a4527d64988ab038632451dff78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:46.109988 8298 raft_consensus.cc:1261] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Refusing update from remote peer b5445a4527d64988ab038632451dff78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:46.110478 8562 consensus_queue.cc:1035] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:46.110668 8513 consensus_queue.cc:1035] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:46.126977 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:46.124021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42713 (local address 127.6.31.131:52897)
0927 10:49:46.124157 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:46.124173 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:46.124252 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:46.124476 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:46.124479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:46.124486 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:49:46.124579 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:46.124585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.125297 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.125302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:46.125864 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:46.125872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.126313 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.126317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:46.126411 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:46.126444 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:49:46.126714 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:49:46.126722 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:49:46.126785 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 10:49:46.126863 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:49:46.126977 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:46.124115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:52897 (local address 127.6.31.129:42713)
0927 10:49:46.124209 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:46.124213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:46.124230 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:49:46.124345 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:46.124347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:46.124382 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:46.124465 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:46.124687 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.125191 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.125979 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:46.126205 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:46.126313 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:46.126557 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:46.126631 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:46.126710 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:46.126793 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 10:49:46.126837 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 10:49:46.127698 8536 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:49:47.687304 8001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:47.687464 8001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:47.690093 8001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f: Sending DeleteTablet for 3 replicas of tablet f04eb2451e7b458abad5621a7c15e661
I20260927 10:49:47.690624 8138 tablet_service.cc:1514] Processing DeleteTablet for tablet f04eb2451e7b458abad5621a7c15e661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:47 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:34456
I20260927 10:49:47.690693 8418 tablet_service.cc:1514] Processing DeleteTablet for tablet f04eb2451e7b458abad5621a7c15e661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:47 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:58258
I20260927 10:49:47.690786 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet f04eb2451e7b458abad5621a7c15e661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:47 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:33274
I20260927 10:49:47.690969 8591 tablet_replica.cc:330] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: stopping tablet replica
I20260927 10:49:47.691028 8592 tablet_replica.cc:330] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: stopping tablet replica
I20260927 10:49:47.691129 8591 raft_consensus.cc:2229] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:49:47.691166 8592 raft_consensus.cc:2229] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:47.691279 8592 raft_consensus.cc:2258] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:47.691295 8593 tablet_replica.cc:330] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: stopping tablet replica
I20260927 10:49:47.691466 8591 raft_consensus.cc:2258] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:47.692022 8592 ts_tablet_manager.cc:1905] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:47.693452 8593 raft_consensus.cc:2229] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:47.693650 8593 raft_consensus.cc:2258] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:47.694415 8593 ts_tablet_manager.cc:1905] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:47.694754 8591 ts_tablet_manager.cc:1905] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:47.695173 8592 ts_tablet_manager.cc:1918] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:49:47.695250 8592 log.cc:1196] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/f04eb2451e7b458abad5621a7c15e661
I20260927 10:49:47.695513 8592 ts_tablet_manager.cc:1939] T f04eb2451e7b458abad5621a7c15e661 P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting consensus metadata
I20260927 10:49:47.698278 7987 catalog_manager.cc:4841] TS 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069): tablet f04eb2451e7b458abad5621a7c15e661 (table test-table [id=298208d7a9b24a58bb3c3d4b55d78126]) successfully deleted
I20260927 10:49:47.698503 8591 ts_tablet_manager.cc:1918] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:49:47.698580 8591 log.cc:1196] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/f04eb2451e7b458abad5621a7c15e661
I20260927 10:49:47.698849 8591 ts_tablet_manager.cc:1939] T f04eb2451e7b458abad5621a7c15e661 P b5445a4527d64988ab038632451dff78: Deleting consensus metadata
I20260927 10:49:47.698889 8593 ts_tablet_manager.cc:1918] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:49:47.698973 8593 log.cc:1196] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/f04eb2451e7b458abad5621a7c15e661
I20260927 10:49:47.699277 8593 ts_tablet_manager.cc:1939] T f04eb2451e7b458abad5621a7c15e661 P a2300a0bbf8a4cffa903cf48f8083400: Deleting consensus metadata
I20260927 10:49:47.699626 7986 catalog_manager.cc:4841] TS b5445a4527d64988ab038632451dff78 (127.6.31.129:42713): tablet f04eb2451e7b458abad5621a7c15e661 (table test-table [id=298208d7a9b24a58bb3c3d4b55d78126]) successfully deleted
I20260927 10:49:47.699978 7987 catalog_manager.cc:4841] TS a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130:46539): tablet f04eb2451e7b458abad5621a7c15e661 (table test-table [id=298208d7a9b24a58bb3c3d4b55d78126]) successfully deleted
Sep 27 10:49:47 dist-test-slave-c60x krb5kdc[7956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506187, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:49:47 dist-test-slave-c60x krb5kdc[7956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506187, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:47.724903 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:47.710720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.6.31.190:33397)
0927 10:49:47.710990 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:47.710996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:47.711010 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:49:47.711075 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:47.711112 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:47.711164 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:47.711245 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:47.712005 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:47.712539 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:47.713378 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:47.713603 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:47.713681 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:47.721042 (+ 7361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:47.721046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:47.721048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:47.721075 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:47.722876 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:47.723437 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:47.723441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:47.723443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:47.723534 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:47.723919 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:47.723922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:47.723923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:47.724119 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:47.724230 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:47.724487 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 10:49:47.724653 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
I20260927 10:49:47.727747 8001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:49:47.728204 8001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:49:47.732949 8276 tablet_service.cc:1467] Processing CreateTablet for tablet da7dd7d295e64df8a5b5c1daf67203fd (DEFAULT_TABLE table=test-table [id=eaeeb65593d047d49dedbc6a5207dfc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:47.732980 8138 tablet_service.cc:1467] Processing CreateTablet for tablet da7dd7d295e64df8a5b5c1daf67203fd (DEFAULT_TABLE table=test-table [id=eaeeb65593d047d49dedbc6a5207dfc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:47.733136 8418 tablet_service.cc:1467] Processing CreateTablet for tablet da7dd7d295e64df8a5b5c1daf67203fd (DEFAULT_TABLE table=test-table [id=eaeeb65593d047d49dedbc6a5207dfc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:47.733263 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7dd7d295e64df8a5b5c1daf67203fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:47.733263 8138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7dd7d295e64df8a5b5c1daf67203fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:47.733419 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7dd7d295e64df8a5b5c1daf67203fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:47.735532 8602 tablet_bootstrap.cc:492] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: Bootstrap starting.
I20260927 10:49:47.735908 8603 tablet_bootstrap.cc:492] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: Bootstrap starting.
I20260927 10:49:47.736553 8602 tablet_bootstrap.cc:654] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:47.736732 8603 tablet_bootstrap.cc:654] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:47.737313 8604 tablet_bootstrap.cc:492] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: Bootstrap starting.
I20260927 10:49:47.737648 8602 tablet_bootstrap.cc:492] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: No bootstrap required, opened a new log
I20260927 10:49:47.737751 8602 ts_tablet_manager.cc:1397] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:47.738102 8604 tablet_bootstrap.cc:654] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:47.738111 8602 raft_consensus.cc:348] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:47.738267 8602 raft_consensus.cc:374] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:47.738310 8602 raft_consensus.cc:729] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe85a810a1b4c718f64b1d60debc2f8, State: Initialized, Role: FOLLOWER
I20260927 10:49:47.738451 8602 consensus_queue.cc:260] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:47.738739 8602 ts_tablet_manager.cc:1428] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:49:47.739514 8604 tablet_bootstrap.cc:492] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: No bootstrap required, opened a new log
I20260927 10:49:47.739533 8603 tablet_bootstrap.cc:492] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: No bootstrap required, opened a new log
I20260927 10:49:47.739581 8604 ts_tablet_manager.cc:1397] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:47.739614 8603 ts_tablet_manager.cc:1397] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 10:49:47.739873 8604 raft_consensus.cc:348] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:47.739950 8604 raft_consensus.cc:374] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:47.739970 8604 raft_consensus.cc:729] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5445a4527d64988ab038632451dff78, State: Initialized, Role: FOLLOWER
I20260927 10:49:47.740026 8603 raft_consensus.cc:348] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:47.740126 8603 raft_consensus.cc:374] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:47.740094 8604 consensus_queue.cc:260] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:47.740163 8603 raft_consensus.cc:729] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2300a0bbf8a4cffa903cf48f8083400, State: Initialized, Role: FOLLOWER
I20260927 10:49:47.740289 8603 consensus_queue.cc:260] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:47.740679 8603 ts_tablet_manager.cc:1428] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:47.741005 8604 ts_tablet_manager.cc:1428] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:47.960307 8608 raft_consensus.cc:482] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:47.960469 8608 raft_consensus.cc:504] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:47.960955 8608 leader_election.cc:288] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5445a4527d64988ab038632451dff78 (127.6.31.129:42713), 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069)
I20260927 10:49:47.961475 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7dd7d295e64df8a5b5c1daf67203fd" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" is_pre_election: true
I20260927 10:49:47.961496 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7dd7d295e64df8a5b5c1daf67203fd" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5445a4527d64988ab038632451dff78" is_pre_election: true
I20260927 10:49:47.961649 8438 raft_consensus.cc:2454] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 0.
I20260927 10:49:47.961673 8158 raft_consensus.cc:2454] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 0.
I20260927 10:49:47.962023 8231 leader_election.cc:302] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe85a810a1b4c718f64b1d60debc2f8, a2300a0bbf8a4cffa903cf48f8083400; no voters:
I20260927 10:49:47.962184 8608 raft_consensus.cc:2781] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:49:47.962245 8608 raft_consensus.cc:482] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:49:47.962309 8608 raft_consensus.cc:3037] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:47.963445 8608 raft_consensus.cc:504] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:47.963769 8608 leader_election.cc:288] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 election: Requested vote from peers b5445a4527d64988ab038632451dff78 (127.6.31.129:42713), 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069)
I20260927 10:49:47.964102 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7dd7d295e64df8a5b5c1daf67203fd" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe85a810a1b4c718f64b1d60debc2f8"
I20260927 10:49:47.964098 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7dd7d295e64df8a5b5c1daf67203fd" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5445a4527d64988ab038632451dff78"
I20260927 10:49:47.964215 8438 raft_consensus.cc:3037] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:47.964215 8158 raft_consensus.cc:3037] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:47.965291 8438 raft_consensus.cc:2454] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 1.
I20260927 10:49:47.965302 8158 raft_consensus.cc:2454] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 1.
I20260927 10:49:47.965584 8231 leader_election.cc:302] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe85a810a1b4c718f64b1d60debc2f8, a2300a0bbf8a4cffa903cf48f8083400; no voters:
I20260927 10:49:47.965767 8608 raft_consensus.cc:2781] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:47.966037 8608 raft_consensus.cc:686] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 1 LEADER]: Becoming Leader. State: Replica: a2300a0bbf8a4cffa903cf48f8083400, State: Running, Role: LEADER
I20260927 10:49:47.966189 8608 consensus_queue.cc:237] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:47.967859 8001 catalog_manager.cc:5495] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "a2300a0bbf8a4cffa903cf48f8083400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } health_report { overall_health: UNKNOWN } } }
I20260927 10:49:47.983174 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:47.979959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.6.31.130:46539)
0927 10:49:47.980220 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:47.980228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:47.980238 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:49:47.980272 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:47.980274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:47.980313 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:47.980406 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:47.980677 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:47.981203 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:47.982057 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:47.982281 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:47.982388 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:47.982668 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:47.982737 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:47.982817 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:47.982907 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:49:47.982966 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260927 10:49:47.989485 8438 raft_consensus.cc:1261] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Refusing update from remote peer a2300a0bbf8a4cffa903cf48f8083400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:47.989496 8158 raft_consensus.cc:1261] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Refusing update from remote peer a2300a0bbf8a4cffa903cf48f8083400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:47.989859 8609 consensus_queue.cc:1035] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:47.990031 8608 consensus_queue.cc:1035] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:48.021281 8000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:48.021512 8000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:48.024156 8000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f: Sending DeleteTablet for 3 replicas of tablet da7dd7d295e64df8a5b5c1daf67203fd
I20260927 10:49:48.024878 8138 tablet_service.cc:1514] Processing DeleteTablet for tablet da7dd7d295e64df8a5b5c1daf67203fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:48 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:34456
I20260927 10:49:48.024940 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet da7dd7d295e64df8a5b5c1daf67203fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:48 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:33274
I20260927 10:49:48.025117 8593 tablet_replica.cc:330] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: stopping tablet replica
I20260927 10:49:48.025250 8593 raft_consensus.cc:2229] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:49:48.025336 8591 tablet_replica.cc:330] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: stopping tablet replica
I20260927 10:49:48.025470 8593 raft_consensus.cc:2258] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:48.025480 8591 raft_consensus.cc:2229] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:48.025602 8591 raft_consensus.cc:2258] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:48.026012 8418 tablet_service.cc:1514] Processing DeleteTablet for tablet da7dd7d295e64df8a5b5c1daf67203fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:48 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:58258
I20260927 10:49:48.026176 8592 tablet_replica.cc:330] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: stopping tablet replica
I20260927 10:49:48.026317 8592 raft_consensus.cc:2229] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:48.026381 8593 ts_tablet_manager.cc:1905] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:48.026427 8592 raft_consensus.cc:2258] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:48.027207 8592 ts_tablet_manager.cc:1905] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:48.027410 8591 ts_tablet_manager.cc:1905] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:48.029075 8593 ts_tablet_manager.cc:1918] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:49:48.029138 8593 log.cc:1196] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/da7dd7d295e64df8a5b5c1daf67203fd
I20260927 10:49:48.029384 8593 ts_tablet_manager.cc:1939] T da7dd7d295e64df8a5b5c1daf67203fd P a2300a0bbf8a4cffa903cf48f8083400: Deleting consensus metadata
I20260927 10:49:48.029702 8000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:49:48.029856 8592 ts_tablet_manager.cc:1918] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:49:48.029927 8592 log.cc:1196] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/da7dd7d295e64df8a5b5c1daf67203fd
I20260927 10:49:48.029979 8591 ts_tablet_manager.cc:1918] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:49:48.030042 8591 log.cc:1196] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/da7dd7d295e64df8a5b5c1daf67203fd
I20260927 10:49:48.030150 8592 ts_tablet_manager.cc:1939] T da7dd7d295e64df8a5b5c1daf67203fd P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting consensus metadata
W20260927 10:49:48.030180 8000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:49:48.030303 8591 ts_tablet_manager.cc:1939] T da7dd7d295e64df8a5b5c1daf67203fd P b5445a4527d64988ab038632451dff78: Deleting consensus metadata
I20260927 10:49:48.030789 7987 catalog_manager.cc:4841] TS a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130:46539): tablet da7dd7d295e64df8a5b5c1daf67203fd (table test-table [id=eaeeb65593d047d49dedbc6a5207dfc9]) successfully deleted
I20260927 10:49:48.030930 7987 catalog_manager.cc:4841] TS 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069): tablet da7dd7d295e64df8a5b5c1daf67203fd (table test-table [id=eaeeb65593d047d49dedbc6a5207dfc9]) successfully deleted
I20260927 10:49:48.031055 7986 catalog_manager.cc:4841] TS b5445a4527d64988ab038632451dff78 (127.6.31.129:42713): tablet da7dd7d295e64df8a5b5c1daf67203fd (table test-table [id=eaeeb65593d047d49dedbc6a5207dfc9]) successfully deleted
I20260927 10:49:48.034963 8138 tablet_service.cc:1467] Processing CreateTablet for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce (DEFAULT_TABLE table=test-table [id=488578e0f1164a61aa989da1a9dddeb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:48.035111 8276 tablet_service.cc:1467] Processing CreateTablet for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce (DEFAULT_TABLE table=test-table [id=488578e0f1164a61aa989da1a9dddeb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:48.035087 8418 tablet_service.cc:1467] Processing CreateTablet for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce (DEFAULT_TABLE table=test-table [id=488578e0f1164a61aa989da1a9dddeb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:49:48.035241 8138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:48.035337 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:48.035336 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:48.037452 8603 tablet_bootstrap.cc:492] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: Bootstrap starting.
I20260927 10:49:48.037479 8604 tablet_bootstrap.cc:492] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78: Bootstrap starting.
I20260927 10:49:48.038307 8604 tablet_bootstrap.cc:654] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:48.038334 8603 tablet_bootstrap.cc:654] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:48.038837 8602 tablet_bootstrap.cc:492] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: Bootstrap starting.
I20260927 10:49:48.039400 8603 tablet_bootstrap.cc:492] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: No bootstrap required, opened a new log
I20260927 10:49:48.039400 8604 tablet_bootstrap.cc:492] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78: No bootstrap required, opened a new log
I20260927 10:49:48.039479 8604 ts_tablet_manager.cc:1397] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:48.039479 8603 ts_tablet_manager.cc:1397] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:48.039549 8602 tablet_bootstrap.cc:654] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:48.039835 8603 raft_consensus.cc:348] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:48.039836 8604 raft_consensus.cc:348] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:48.039935 8604 raft_consensus.cc:374] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:48.039935 8603 raft_consensus.cc:374] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:48.039969 8603 raft_consensus.cc:729] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2300a0bbf8a4cffa903cf48f8083400, State: Initialized, Role: FOLLOWER
I20260927 10:49:48.039969 8604 raft_consensus.cc:729] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5445a4527d64988ab038632451dff78, State: Initialized, Role: FOLLOWER
I20260927 10:49:48.040086 8604 consensus_queue.cc:260] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:48.040086 8603 consensus_queue.cc:260] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:48.040343 8603 ts_tablet_manager.cc:1428] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:48.040343 8604 ts_tablet_manager.cc:1428] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:49:48.040499 8602 tablet_bootstrap.cc:492] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: No bootstrap required, opened a new log
I20260927 10:49:48.040580 8602 ts_tablet_manager.cc:1397] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:48.040975 8602 raft_consensus.cc:348] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:48.041116 8602 raft_consensus.cc:374] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:48.041162 8602 raft_consensus.cc:729] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe85a810a1b4c718f64b1d60debc2f8, State: Initialized, Role: FOLLOWER
I20260927 10:49:48.041290 8602 consensus_queue.cc:260] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:48.041575 8602 ts_tablet_manager.cc:1428] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:49:48.321632 8609 raft_consensus.cc:482] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:49:48.321864 8609 raft_consensus.cc:504] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:48.322366 8609 leader_election.cc:288] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5445a4527d64988ab038632451dff78 (127.6.31.129:42713), 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069)
I20260927 10:49:48.322880 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f83a0dfb8f4b00a3c9a6e93fee85ce" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" is_pre_election: true
I20260927 10:49:48.322908 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f83a0dfb8f4b00a3c9a6e93fee85ce" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5445a4527d64988ab038632451dff78" is_pre_election: true
I20260927 10:49:48.323040 8438 raft_consensus.cc:2454] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 0.
I20260927 10:49:48.323133 8158 raft_consensus.cc:2454] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 0.
I20260927 10:49:48.323380 8231 leader_election.cc:302] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe85a810a1b4c718f64b1d60debc2f8, a2300a0bbf8a4cffa903cf48f8083400; no voters:
I20260927 10:49:48.323551 8609 raft_consensus.cc:2781] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:49:48.323609 8609 raft_consensus.cc:482] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:49:48.323647 8609 raft_consensus.cc:3037] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:48.324607 8609 raft_consensus.cc:504] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:48.324958 8609 leader_election.cc:288] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 election: Requested vote from peers b5445a4527d64988ab038632451dff78 (127.6.31.129:42713), 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069)
I20260927 10:49:48.325286 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f83a0dfb8f4b00a3c9a6e93fee85ce" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe85a810a1b4c718f64b1d60debc2f8"
I20260927 10:49:48.325284 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f83a0dfb8f4b00a3c9a6e93fee85ce" candidate_uuid: "a2300a0bbf8a4cffa903cf48f8083400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5445a4527d64988ab038632451dff78"
I20260927 10:49:48.325392 8158 raft_consensus.cc:3037] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:48.325412 8438 raft_consensus.cc:3037] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:48.326511 8438 raft_consensus.cc:2454] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 1.
I20260927 10:49:48.326557 8158 raft_consensus.cc:2454] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2300a0bbf8a4cffa903cf48f8083400 in term 1.
I20260927 10:49:48.326805 8231 leader_election.cc:302] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe85a810a1b4c718f64b1d60debc2f8, a2300a0bbf8a4cffa903cf48f8083400; no voters:
I20260927 10:49:48.326974 8609 raft_consensus.cc:2781] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:48.327080 8609 raft_consensus.cc:686] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 1 LEADER]: Becoming Leader. State: Replica: a2300a0bbf8a4cffa903cf48f8083400, State: Running, Role: LEADER
I20260927 10:49:48.327231 8609 consensus_queue.cc:237] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } }
I20260927 10:49:48.328881 8000 catalog_manager.cc:5495] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "a2300a0bbf8a4cffa903cf48f8083400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2300a0bbf8a4cffa903cf48f8083400" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 } health_report { overall_health: UNKNOWN } } }
I20260927 10:49:48.411557 8438 raft_consensus.cc:1261] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Refusing update from remote peer a2300a0bbf8a4cffa903cf48f8083400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:48.411671 8158 raft_consensus.cc:1261] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P b5445a4527d64988ab038632451dff78 [term 1 FOLLOWER]: Refusing update from remote peer a2300a0bbf8a4cffa903cf48f8083400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:48.412103 8609 consensus_queue.cc:1035] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe85a810a1b4c718f64b1d60debc2f8" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:48.412298 8616 consensus_queue.cc:1035] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5445a4527d64988ab038632451dff78" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:49:48.425870 8575 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:49:49.974675 8000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:49.974857 8000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:49:49.977342 8000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3b7bf511b404287a0724eaf96fa595f: Sending DeleteTablet for 3 replicas of tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce
I20260927 10:49:49.978009 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:49 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:33274
I20260927 10:49:49.977998 8138 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:49 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:34456
I20260927 10:49:49.978005 8418 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:49:49 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:58258
I20260927 10:49:49.978349 8644 tablet_replica.cc:330] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: stopping tablet replica
I20260927 10:49:49.978403 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 8073
I20260927 10:49:49.978500 8644 raft_consensus.cc:2229] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:49:49.978461 8646 tablet_replica.cc:330] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: stopping tablet replica
I20260927 10:49:49.979503 8646 raft_consensus.cc:2229] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:49:49.979524 8644 raft_consensus.cc:2258] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:49.979609 8646 raft_consensus.cc:2258] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:49:49.980157 8644 ts_tablet_manager.cc:1905] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:49.980398 8646 ts_tablet_manager.cc:1905] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:49:49.982888 8644 ts_tablet_manager.cc:1918] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:49:49.982987 8644 log.cc:1196] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/b3f83a0dfb8f4b00a3c9a6e93fee85ce
I20260927 10:49:49.983016 8646 ts_tablet_manager.cc:1918] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:49:49.983105 8646 log.cc:1196] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/b3f83a0dfb8f4b00a3c9a6e93fee85ce
I20260927 10:49:49.983328 8644 ts_tablet_manager.cc:1939] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P a2300a0bbf8a4cffa903cf48f8083400: Deleting consensus metadata
I20260927 10:49:49.983352 8646 ts_tablet_manager.cc:1939] T b3f83a0dfb8f4b00a3c9a6e93fee85ce P 2fe85a810a1b4c718f64b1d60debc2f8: Deleting consensus metadata
I20260927 10:49:49.984107 7987 catalog_manager.cc:4841] TS a2300a0bbf8a4cffa903cf48f8083400 (127.6.31.130:46539): tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce (table test-table [id=488578e0f1164a61aa989da1a9dddeb2]) successfully deleted
I20260927 10:49:49.984297 7987 catalog_manager.cc:4841] TS 2fe85a810a1b4c718f64b1d60debc2f8 (127.6.31.131:33069): tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce (table test-table [id=488578e0f1164a61aa989da1a9dddeb2]) successfully deleted
I20260927 10:49:49.986593 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 8213
W20260927 10:49:49.986603 7986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:42713 (error 108)
I20260927 10:49:49.988821 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:49.988285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42713 (local address 127.0.0.1:34466)
0927 10:49:49.988575 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:49.988667 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
W20260927 10:49:49.988969 7986 catalog_manager.cc:4584] TS b5445a4527d64988ab038632451dff78 (127.6.31.129:42713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3f83a0dfb8f4b00a3c9a6e93fee85ce: Network error: Client connection negotiation failed: client connection to 127.6.31.129:42713: connect: Connection refused (error 111)
I20260927 10:49:49.993528 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 8353
I20260927 10:49:50.003603 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 7969
2026-09-27T10:49:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6890 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:50 dist-test-slave-c60x krb5kdc[8651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:50 dist-test-slave-c60x krb5kdc[8651](info): set up 2 sockets
Sep 27 10:49:50 dist-test-slave-c60x krb5kdc[8651](info): commencing operation
krb5kdc: starting...
W20260927 10:49:51.051157 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:49:51 dist-test-slave-c60x krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:49:51.179606 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:45811
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:45811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:51.291242 8664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:51.291518 8664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:51.291559 8664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:51.295581 8664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:51.295643 8664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:51.295662 8664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:51.295679 8664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:51.295696 8664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:51.300972 8664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:45811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:51.301997 8664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:51.302990 8664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:51.312335 8669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:51.312337 8670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:51.312337 8672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:51.312379 8664 server_base.cc:1034] running on GCE node
I20260927 10:49:51.313216 8664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:51.314366 8664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:51.315533 8664 hybrid_clock.cc:648] HybridClock initialized: now 1790506191315505 us; error 43 us; skew 500 ppm
Sep 27 10:49:51 dist-test-slave-c60x krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506191, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:51.318378 8664 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:49:51.319372 8664 webserver.cc:460] Webserver started at http://127.6.31.190:37605/ using document root <none> and password file <none>
I20260927 10:49:51.319970 8664 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:51.320070 8664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:51.320305 8664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:51.322333 8664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "2e02f10a0e4c47718cf075fad454aa70"
format_stamp: "Formatted at 2026-09-27 10:49:51 on dist-test-slave-c60x"
server_key: "98980279ae958461bce1da2a195cb8a6"
server_key_iv: "2aec8d6fca752f1a2d7071efee7997d0"
server_key_version: "encryptionkey@0"
I20260927 10:49:51.322921 8664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "2e02f10a0e4c47718cf075fad454aa70"
format_stamp: "Formatted at 2026-09-27 10:49:51 on dist-test-slave-c60x"
server_key: "98980279ae958461bce1da2a195cb8a6"
server_key_iv: "2aec8d6fca752f1a2d7071efee7997d0"
server_key_version: "encryptionkey@0"
I20260927 10:49:51.326556 8664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:49:51.329034 8679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:51.330404 8664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:51.330572 8664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "2e02f10a0e4c47718cf075fad454aa70"
format_stamp: "Formatted at 2026-09-27 10:49:51 on dist-test-slave-c60x"
server_key: "98980279ae958461bce1da2a195cb8a6"
server_key_iv: "2aec8d6fca752f1a2d7071efee7997d0"
server_key_version: "encryptionkey@0"
I20260927 10:49:51.330694 8664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:51.355064 8664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:51.358280 8664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:51.358520 8664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:51.366026 8664 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:45811
I20260927 10:49:51.366051 8731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:45811 every 8 connection(s)
I20260927 10:49:51.367044 8664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:51.369977 8732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:51.376008 8732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70: Bootstrap starting.
I20260927 10:49:51.376420 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 8664
I20260927 10:49:51.376515 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:49:51.376791 6270 external_mini_cluster.cc:1428] Setting key b2b2285384bfae4b96cbf0003376928c
I20260927 10:49:51.378782 8732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:51.379598 8732 log.cc:824] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:51.381464 8732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70: No bootstrap required, opened a new log
I20260927 10:49:51.384930 8732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e02f10a0e4c47718cf075fad454aa70" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 45811 } }
I20260927 10:49:51.385172 8732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:51.385229 8732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e02f10a0e4c47718cf075fad454aa70, State: Initialized, Role: FOLLOWER
Sep 27 10:49:51 dist-test-slave-c60x krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:51.385792 8732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e02f10a0e4c47718cf075fad454aa70" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 45811 } }
I20260927 10:49:51.385982 8732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:51.386075 8732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:51.386176 8732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:51.387490 8732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e02f10a0e4c47718cf075fad454aa70" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 45811 } }
I20260927 10:49:51.387871 8732 leader_election.cc:302] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e02f10a0e4c47718cf075fad454aa70; no voters:
I20260927 10:49:51.388223 8732 leader_election.cc:288] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:51.388345 8737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:51.388581 8737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [term 1 LEADER]: Becoming Leader. State: Replica: 2e02f10a0e4c47718cf075fad454aa70, State: Running, Role: LEADER
I20260927 10:49:51.388887 8737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e02f10a0e4c47718cf075fad454aa70" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 45811 } }
I20260927 10:49:51.389607 8732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:51.390537 8739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e02f10a0e4c47718cf075fad454aa70. Latest consensus state: current_term: 1 leader_uuid: "2e02f10a0e4c47718cf075fad454aa70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e02f10a0e4c47718cf075fad454aa70" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 45811 } } }
I20260927 10:49:51.390666 8739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:51.390992 8738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e02f10a0e4c47718cf075fad454aa70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e02f10a0e4c47718cf075fad454aa70" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 45811 } } }
I20260927 10:49:51.391116 8738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:51.391496 8746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:51.392617 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:51.378234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.6.31.190:45811)
0927 10:49:51.378694 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:51.378709 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:51.378743 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:49:51.379580 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:51.379601 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:51.379991 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:51.380323 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:51.381120 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.382137 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.383125 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.383597 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.383769 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:51.386430 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:51.386437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:51.386455 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:51.386490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:51.388494 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:51.389020 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:51.389026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:51.389032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:51.389161 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:51.389514 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:51.389517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:51.389519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:51.390048 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:51.390966 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:51.391141 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 10:49:51.391488 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":129,"threads_started":1}
I20260927 10:49:51.394639 8746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:51.400218 8746 catalog_manager.cc:1347] Generated new cluster ID: cfff4894ae3f444d8f50a76561925e35
I20260927 10:49:51.400295 8746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:51.409142 8746 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:51.410115 8746 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:51.419111 8746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e02f10a0e4c47718cf075fad454aa70: Generated new TSK 0
I20260927 10:49:51.419809 8746 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:49:51.451246 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:45811
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:51.563581 8758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:51.563828 8758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:51.563872 8758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:51.567852 8758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:51.567904 8758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:51.567960 8758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:49:51.573155 8758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:51.574156 8758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:51.575125 8758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:51.584625 8766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:51.584705 8763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:51.584762 8758 server_base.cc:1034] running on GCE node
W20260927 10:49:51.584725 8764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:51.585415 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:51.586108 8758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:51.587304 8758 hybrid_clock.cc:648] HybridClock initialized: now 1790506191587270 us; error 46 us; skew 500 ppm
Sep 27 10:49:51 dist-test-slave-c60x krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506191, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:51.590245 8758 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:49:51.591197 8758 webserver.cc:460] Webserver started at http://127.6.31.129:42885/ using document root <none> and password file <none>
I20260927 10:49:51.591708 8758 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:51.591764 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:51.592001 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:51.594033 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "c0953c1ccfaa4af4942ebc7a3de42f12"
format_stamp: "Formatted at 2026-09-27 10:49:51 on dist-test-slave-c60x"
server_key: "a26ab42319b658311de0068ad02770a2"
server_key_iv: "c5a42fa9b1176a64c8761024e9e7aebe"
server_key_version: "encryptionkey@0"
I20260927 10:49:51.594527 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "c0953c1ccfaa4af4942ebc7a3de42f12"
format_stamp: "Formatted at 2026-09-27 10:49:51 on dist-test-slave-c60x"
server_key: "a26ab42319b658311de0068ad02770a2"
server_key_iv: "c5a42fa9b1176a64c8761024e9e7aebe"
server_key_version: "encryptionkey@0"
I20260927 10:49:51.598244 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:49:51.600579 8773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:51.601807 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:51.601958 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "c0953c1ccfaa4af4942ebc7a3de42f12"
format_stamp: "Formatted at 2026-09-27 10:49:51 on dist-test-slave-c60x"
server_key: "a26ab42319b658311de0068ad02770a2"
server_key_iv: "c5a42fa9b1176a64c8761024e9e7aebe"
server_key_version: "encryptionkey@0"
I20260927 10:49:51.602061 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:51.617147 8758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:51.620535 8758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:51.620760 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:51.621387 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:51.622241 8758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:51.622287 8758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:51.622329 8758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:51.622390 8758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:51.631484 8758 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:33455
I20260927 10:49:51.631554 8886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:33455 every 8 connection(s)
I20260927 10:49:51.632406 8758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:49:51.637514 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 8758
I20260927 10:49:51.637643 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:49:51.637930 6270 external_mini_cluster.cc:1428] Setting key 88409e09339c721b37ca2ca0fa0d5a88
Sep 27 10:49:51 dist-test-slave-c60x krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506191, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:49:51.647881 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:51.634291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:36421 (local address 127.6.31.190:45811)
0927 10:49:51.634446 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:51.634451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:51.635435 (+ 984us) server_negotiation.cc:407] Connection header received
0927 10:49:51.636927 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:51.636930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:51.637004 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:51.637083 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:51.638902 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.639598 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.640999 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.641242 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.641370 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:51.644142 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:51.644148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:51.644155 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:51.644181 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:51.645725 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:51.646306 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:51.646311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:51.646317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:51.646393 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:51.646752 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:51.646755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:51.646757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:51.646941 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:51.647048 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:51.647592 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 10:49:51.647716 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:49:51.648723 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:51.634756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:45811 (local address 127.6.31.129:36421)
0927 10:49:51.635283 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:51.635328 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:51.636592 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:51.637578 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:51.637588 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:51.638109 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:51.638744 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:51.638763 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.639718 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.639721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:51.640873 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:51.640882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.641411 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.641414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:51.641584 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:51.642345 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:51.642361 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:51.643962 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:51.645903 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:51.645911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:51.645930 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:51.646187 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:51.646501 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:51.646503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:51.646505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:51.646645 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:51.647058 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:51.647067 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:51.647344 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 10:49:51.647541 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 10:49:51.647884 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":98,"threads_started":1}
I20260927 10:49:51.650134 8887 heartbeater.cc:347] Connected to a master server at 127.6.31.190:45811
I20260927 10:49:51.650400 8887 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:49:51.651041 8887 heartbeater.cc:511] Master 127.6.31.190:45811 requested a full tablet report, sending...
I20260927 10:49:51.652716 8696 ts_manager.cc:194] Registered new tserver with Master: c0953c1ccfaa4af4942ebc7a3de42f12 (127.6.31.129:33455)
I20260927 10:49:51.653853 8696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:36421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:49:51.666630 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:45811
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:51.777549 8892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:51.777899 8892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:51.778002 8892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:51.782438 8892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:51.782532 8892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:51.782622 8892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:49:51.788252 8892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:51.789486 8892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:51.790571 8892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:49:51.800364 8892 server_base.cc:1034] running on GCE node
W20260927 10:49:51.800328 8897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:51.800320 8900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:51.800328 8898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:51.800984 8892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:51.801664 8892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:51.802834 8892 hybrid_clock.cc:648] HybridClock initialized: now 1790506191802783 us; error 63 us; skew 500 ppm
Sep 27 10:49:51 dist-test-slave-c60x krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506191, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:51.805917 8892 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:49:51.806955 8892 webserver.cc:460] Webserver started at http://127.6.31.130:35283/ using document root <none> and password file <none>
I20260927 10:49:51.807564 8892 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:51.807626 8892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:51.807859 8892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:51.809894 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "e653e6d101d942f18b4319c5ed969b3e"
format_stamp: "Formatted at 2026-09-27 10:49:51 on dist-test-slave-c60x"
server_key: "938aa6c608f87e3726395d44cf5807d2"
server_key_iv: "33293f9330ce0b39ae44924e305caeec"
server_key_version: "encryptionkey@0"
I20260927 10:49:51.810472 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "e653e6d101d942f18b4319c5ed969b3e"
format_stamp: "Formatted at 2026-09-27 10:49:51 on dist-test-slave-c60x"
server_key: "938aa6c608f87e3726395d44cf5807d2"
server_key_iv: "33293f9330ce0b39ae44924e305caeec"
server_key_version: "encryptionkey@0"
I20260927 10:49:51.814503 8892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 10:49:51.817247 8907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:51.818598 8892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:49:51.818763 8892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "e653e6d101d942f18b4319c5ed969b3e"
format_stamp: "Formatted at 2026-09-27 10:49:51 on dist-test-slave-c60x"
server_key: "938aa6c608f87e3726395d44cf5807d2"
server_key_iv: "33293f9330ce0b39ae44924e305caeec"
server_key_version: "encryptionkey@0"
I20260927 10:49:51.818960 8892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:51.840252 8892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:51.843765 8892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:51.843987 8892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:51.844606 8892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:51.845443 8892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:51.845520 8892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:51.845589 8892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:51.845640 8892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:51.855134 8892 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:40805
I20260927 10:49:51.855172 9020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:40805 every 8 connection(s)
I20260927 10:49:51.856107 8892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:49:51.863400 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 8892
I20260927 10:49:51.863528 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:49:51.863858 6270 external_mini_cluster.cc:1428] Setting key b9a08cec22d2541d0c13776ee5722df8
Sep 27 10:49:51 dist-test-slave-c60x krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506191, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:51.871250 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:51.858024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:52155 (local address 127.6.31.190:45811)
0927 10:49:51.858176 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:51.858181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:51.859129 (+ 948us) server_negotiation.cc:407] Connection header received
0927 10:49:51.860504 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:51.860507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:51.860553 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:51.860631 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:51.861919 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.862452 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.863780 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.864154 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.864286 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:51.867485 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:51.867488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:51.867491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:51.867513 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:51.868995 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:51.869561 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:51.869570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:51.869572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:51.869647 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:51.870069 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:51.870073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:51.870074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:51.870269 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:51.870416 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:51.870916 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 10:49:51.871036 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:49:51.872108 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:51.858426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:45811 (local address 127.6.31.130:52155)
0927 10:49:51.858944 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:51.858988 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:51.860287 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:51.860840 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:51.860849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:51.861251 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:51.861743 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:51.861755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.862586 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.862589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:51.863658 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:51.863672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:51.864376 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:51.864381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:51.864592 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:51.865477 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:51.865499 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:51.867300 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:51.869137 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:51.869143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:51.869159 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:51.869442 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:51.869752 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:51.869755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:51.869758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:51.869955 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:51.870395 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:51.870402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:51.870669 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 10:49:51.870864 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 10:49:51.871254 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":129,"threads_started":1}
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:49:51.873631 9021 heartbeater.cc:347] Connected to a master server at 127.6.31.190:45811
I20260927 10:49:51.873971 9021 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:51.874544 9021 heartbeater.cc:511] Master 127.6.31.190:45811 requested a full tablet report, sending...
I20260927 10:49:51.875906 8696 ts_manager.cc:194] Registered new tserver with Master: e653e6d101d942f18b4319c5ed969b3e (127.6.31.130:40805)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
I20260927 10:49:51.876694 8696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:52155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:49:51.894423 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:45811
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:52.005676 9026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:52.005946 9026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:52.006029 9026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:52.010183 9026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:52.010244 9026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:52.010340 9026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:49:52.016206 9026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:52.017308 9026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:52.018378 9026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:52.030336 9034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:52.030360 9032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:52.030345 9031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:52.030383 9026 server_base.cc:1034] running on GCE node
I20260927 10:49:52.030973 9026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:52.031698 9026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:52.032876 9026 hybrid_clock.cc:648] HybridClock initialized: now 1790506192032852 us; error 29 us; skew 500 ppm
Sep 27 10:49:52 dist-test-slave-c60x krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506192, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:52.035694 9026 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:49:52.036615 9026 webserver.cc:460] Webserver started at http://127.6.31.131:46519/ using document root <none> and password file <none>
I20260927 10:49:52.037206 9026 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:52.037298 9026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:52.037537 9026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:52.039731 9026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "3549a280e24e4b1986fe527e38c8c2a4"
format_stamp: "Formatted at 2026-09-27 10:49:52 on dist-test-slave-c60x"
server_key: "477fe526bd3bc8a1939314e123c1f524"
server_key_iv: "a62732b859531ccae8c3d76571d8fa93"
server_key_version: "encryptionkey@0"
I20260927 10:49:52.040313 9026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "3549a280e24e4b1986fe527e38c8c2a4"
format_stamp: "Formatted at 2026-09-27 10:49:52 on dist-test-slave-c60x"
server_key: "477fe526bd3bc8a1939314e123c1f524"
server_key_iv: "a62732b859531ccae8c3d76571d8fa93"
server_key_version: "encryptionkey@0"
I20260927 10:49:52.044067 9026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:49:52.046540 9041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:52.047834 9026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:52.047977 9026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "3549a280e24e4b1986fe527e38c8c2a4"
format_stamp: "Formatted at 2026-09-27 10:49:52 on dist-test-slave-c60x"
server_key: "477fe526bd3bc8a1939314e123c1f524"
server_key_iv: "a62732b859531ccae8c3d76571d8fa93"
server_key_version: "encryptionkey@0"
I20260927 10:49:52.048112 9026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:52.066143 9026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:52.069100 9026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:52.069312 9026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:52.069901 9026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:52.070721 9026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:52.070802 9026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:52.070899 9026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:52.070957 9026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:52.080096 9026 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:37923
I20260927 10:49:52.080116 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:37923 every 8 connection(s)
I20260927 10:49:52.080963 9026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:49:52 dist-test-slave-c60x krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506192, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:52.091220 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 9026
I20260927 10:49:52.091315 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:49:52.091722 6270 external_mini_cluster.cc:1428] Setting key 6d55cf0c9711e28bb9b93ecb09ebdf0e
I20260927 10:49:52.095258 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:52.082794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:51051 (local address 127.6.31.190:45811)
0927 10:49:52.082934 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:52.082938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:52.083775 (+ 837us) server_negotiation.cc:407] Connection header received
0927 10:49:52.085131 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:52.085135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:52.085191 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:52.085272 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:52.086351 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:52.086890 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:52.088118 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:52.088344 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:52.088462 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:52.091164 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:52.091168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:52.091170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:52.091193 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:52.092985 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:52.093673 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:52.093677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:52.093679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:52.093767 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:52.094110 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:52.094115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:52.094116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:52.094302 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:52.094410 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:52.094887 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 10:49:52.095066 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:49:52.096076 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:52.083218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:45811 (local address 127.6.31.131:51051)
0927 10:49:52.083630 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:52.083671 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:52.084922 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:52.085431 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:52.085438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:52.085768 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:52.086207 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:52.086218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:52.087051 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:52.087055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:52.087986 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:52.087995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:52.088507 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:52.088510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:52.088671 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:52.089226 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:52.089240 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:52.090841 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:52.093249 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:52.093255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:52.093275 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:52.093559 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:52.093867 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:52.093871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:52.093873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:52.094011 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:52.094406 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:52.094412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:52.094655 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 10:49:52.094845 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 10:49:52.095191 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260927 10:49:52.097124 9155 heartbeater.cc:347] Connected to a master server at 127.6.31.190:45811
I20260927 10:49:52.097353 9155 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:52.097833 9155 heartbeater.cc:511] Master 127.6.31.190:45811 requested a full tablet report, sending...
I20260927 10:49:52.098803 8696 ts_manager.cc:194] Registered new tserver with Master: 3549a280e24e4b1986fe527e38c8c2a4 (127.6.31.131:37923)
I20260927 10:49:52.099443 8696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:51051
I20260927 10:49:52.105803 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:49:52.118657 8735 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:52.117005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.6.31.190:45811)
0927 10:49:52.117190 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:52.117196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:52.117381 (+ 185us) server_negotiation.cc:407] Connection header received
0927 10:49:52.117567 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:52.117571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:52.117624 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:52.117717 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:52.118497 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38868: BlockingRecv error: recv got EOF from 127.0.0.1:38868 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 10:49:52.119767 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 8758
I20260927 10:49:52.125361 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 8892
I20260927 10:49:52.130594 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 9026
I20260927 10:49:52.136221 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 8664
2026-09-27T10:49:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2129 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:52 dist-test-slave-c60x krb5kdc[9167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:52 dist-test-slave-c60x krb5kdc[9167](info): set up 2 sockets
Sep 27 10:49:52 dist-test-slave-c60x krb5kdc[9167](info): commencing operation
krb5kdc: starting...
W20260927 10:49:53.187037 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:49:53 dist-test-slave-c60x krb5kdc[9167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:49:53.310846 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35025
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:35025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:53.425518 9180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:53.425880 9180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:53.425979 9180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:53.430596 9180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:53.430747 9180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:53.430794 9180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:53.430841 9180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:53.430894 9180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:53.437201 9180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:35025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:53.438465 9180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:53.439635 9180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:53.448273 9186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:53.448272 9188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:53.448410 9180 server_base.cc:1034] running on GCE node
W20260927 10:49:53.448366 9185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:53.449121 9180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:53.450292 9180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:53.451469 9180 hybrid_clock.cc:648] HybridClock initialized: now 1790506193451438 us; error 42 us; skew 500 ppm
Sep 27 10:49:53 dist-test-slave-c60x krb5kdc[9167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506193, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:53.454239 9180 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:49:53.455212 9180 webserver.cc:460] Webserver started at http://127.6.31.190:35091/ using document root <none> and password file <none>
I20260927 10:49:53.455783 9180 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:53.455889 9180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:53.456121 9180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:53.458217 9180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "ae34239ce1a34f499a4bdcb1482ec479"
format_stamp: "Formatted at 2026-09-27 10:49:53 on dist-test-slave-c60x"
server_key: "aa2c8d7f14e57a8ca75c4a9242cfc118"
server_key_iv: "8d93bd07a7f8ceb50d615f8e326fae88"
server_key_version: "encryptionkey@0"
I20260927 10:49:53.458791 9180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "ae34239ce1a34f499a4bdcb1482ec479"
format_stamp: "Formatted at 2026-09-27 10:49:53 on dist-test-slave-c60x"
server_key: "aa2c8d7f14e57a8ca75c4a9242cfc118"
server_key_iv: "8d93bd07a7f8ceb50d615f8e326fae88"
server_key_version: "encryptionkey@0"
I20260927 10:49:53.462416 9180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:49:53.464829 9195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:53.465999 9180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:49:53.466138 9180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "ae34239ce1a34f499a4bdcb1482ec479"
format_stamp: "Formatted at 2026-09-27 10:49:53 on dist-test-slave-c60x"
server_key: "aa2c8d7f14e57a8ca75c4a9242cfc118"
server_key_iv: "8d93bd07a7f8ceb50d615f8e326fae88"
server_key_version: "encryptionkey@0"
I20260927 10:49:53.466253 9180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:53.478057 9180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:53.481241 9180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:53.481458 9180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:53.488777 9180 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:35025
I20260927 10:49:53.489099 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:35025 every 8 connection(s)
I20260927 10:49:53.489789 9180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:53.492686 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:53.497494 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 9180
I20260927 10:49:53.497648 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:49:53.497916 6270 external_mini_cluster.cc:1428] Setting key 8006a7553ecf50a68d7660b868e5eb32
I20260927 10:49:53.498454 9248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479: Bootstrap starting.
I20260927 10:49:53.501796 9248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:53.502700 9248 log.cc:824] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:53.504987 9248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479: No bootstrap required, opened a new log
Sep 27 10:49:53 dist-test-slave-c60x krb5kdc[9167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:53.509527 9248 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34239ce1a34f499a4bdcb1482ec479" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35025 } }
I20260927 10:49:53.509786 9248 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:53.509865 9248 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae34239ce1a34f499a4bdcb1482ec479, State: Initialized, Role: FOLLOWER
I20260927 10:49:53.510341 9248 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34239ce1a34f499a4bdcb1482ec479" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35025 } }
I20260927 10:49:53.510495 9248 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:53.510579 9248 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:53.510679 9248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:53.511667 9248 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34239ce1a34f499a4bdcb1482ec479" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35025 } }
I20260927 10:49:53.512020 9248 leader_election.cc:302] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae34239ce1a34f499a4bdcb1482ec479; no voters:
I20260927 10:49:53.512373 9248 leader_election.cc:288] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:53.512483 9253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:53.512698 9253 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [term 1 LEADER]: Becoming Leader. State: Replica: ae34239ce1a34f499a4bdcb1482ec479, State: Running, Role: LEADER
I20260927 10:49:53.513079 9253 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34239ce1a34f499a4bdcb1482ec479" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35025 } }
I20260927 10:49:53.513528 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:53.499316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.6.31.190:35025)
0927 10:49:53.499830 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:53.499846 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:53.499897 (+ 51us) server_negotiation.cc:407] Connection header received
0927 10:49:53.500649 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:53.500667 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:53.501037 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:53.501382 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:53.502426 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:53.503520 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:53.504671 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:53.505241 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:53.505378 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:53.508311 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:53.508316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:53.508330 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:53.508358 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:53.510247 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:53.510760 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:53.510769 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:53.510775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:53.510906 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:53.511246 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:53.511251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:53.511253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:53.511671 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:53.511877 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:53.512177 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 10:49:53.512490 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":156,"threads_started":1}
I20260927 10:49:53.513757 9248 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:53.515596 9255 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae34239ce1a34f499a4bdcb1482ec479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34239ce1a34f499a4bdcb1482ec479" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35025 } } }
I20260927 10:49:53.515731 9255 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:53.516568 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:53.516040 9254 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae34239ce1a34f499a4bdcb1482ec479. Latest consensus state: current_term: 1 leader_uuid: "ae34239ce1a34f499a4bdcb1482ec479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34239ce1a34f499a4bdcb1482ec479" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35025 } } }
I20260927 10:49:53.517496 9254 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479 [sys.catalog]: This master's current role is: LEADER
W20260927 10:49:53.518926 9267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:53.519022 9267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:53.519114 9268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:53.520033 9268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:53.527060 9268 catalog_manager.cc:1347] Generated new cluster ID: 9b4fb012bf244e4f9326ca5b0c72e8e2
I20260927 10:49:53.527163 9268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:53.537269 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:53.531542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.6.31.190:35025)
0927 10:49:53.531655 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:53.531660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:53.531992 (+ 332us) server_negotiation.cc:407] Connection header received
0927 10:49:53.532133 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:53.532136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:53.532183 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:53.532263 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:53.532867 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:53.533629 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:53.534793 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:53.535182 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:53.535270 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:53.536229 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:53.536233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:53.536236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:53.536801 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:53.536834 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:53.536852 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:53.536936 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:53.537067 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 10:49:53.537124 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:49:53.546186 9268 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:53.547483 9268 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:53.555056 9268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae34239ce1a34f499a4bdcb1482ec479: Generated new TSK 0
I20260927 10:49:53.555918 9268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:53.566207 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 9180
2026-09-27T10:49:53Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1427 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:53 dist-test-slave-c60x krb5kdc[9281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:53 dist-test-slave-c60x krb5kdc[9281](info): set up 2 sockets
Sep 27 10:49:53 dist-test-slave-c60x krb5kdc[9281](info): commencing operation
krb5kdc: starting...
W20260927 10:49:54.615482 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:49:54 dist-test-slave-c60x krb5kdc[9281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:49:54.743240 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:43869
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:42095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:43869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:54.853128 9294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:54.853400 9294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:54.853441 9294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:54.857551 9294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:54.857617 9294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:54.857636 9294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:54.857653 9294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:54.857668 9294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:54.862967 9294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:43869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:43869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:54.864008 9294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:54.864965 9294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:54.873461 9299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:54.873488 9300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:54.873572 9294 server_base.cc:1034] running on GCE node
W20260927 10:49:54.873504 9302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:54.874390 9294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:54.875563 9294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:54.876739 9294 hybrid_clock.cc:648] HybridClock initialized: now 1790506194876725 us; error 26 us; skew 500 ppm
Sep 27 10:49:54 dist-test-slave-c60x krb5kdc[9281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506194, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:54.879679 9294 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:49:54.880585 9294 webserver.cc:460] Webserver started at http://127.6.31.190:34841/ using document root <none> and password file <none>
I20260927 10:49:54.881139 9294 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:54.881197 9294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:54.881420 9294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:54.883512 9294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "5e63d4f28c58463082347e3e4bc69aeb"
format_stamp: "Formatted at 2026-09-27 10:49:54 on dist-test-slave-c60x"
server_key: "45616858d75c4b06f00841857adc8ade"
server_key_iv: "a93ca5671d13fb9952107d73e0c44673"
server_key_version: "encryptionkey@0"
I20260927 10:49:54.884090 9294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "5e63d4f28c58463082347e3e4bc69aeb"
format_stamp: "Formatted at 2026-09-27 10:49:54 on dist-test-slave-c60x"
server_key: "45616858d75c4b06f00841857adc8ade"
server_key_iv: "a93ca5671d13fb9952107d73e0c44673"
server_key_version: "encryptionkey@0"
I20260927 10:49:54.887871 9294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:49:54.890300 9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:54.891613 9294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:54.891739 9294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "5e63d4f28c58463082347e3e4bc69aeb"
format_stamp: "Formatted at 2026-09-27 10:49:54 on dist-test-slave-c60x"
server_key: "45616858d75c4b06f00841857adc8ade"
server_key_iv: "a93ca5671d13fb9952107d73e0c44673"
server_key_version: "encryptionkey@0"
I20260927 10:49:54.891846 9294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:54.906860 9294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:54.910584 9294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:54.910827 9294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:54.918156 9294 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:43869
I20260927 10:49:54.918458 9361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:43869 every 8 connection(s)
I20260927 10:49:54.919265 9294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:54.922248 9362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:54.928079 9362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb: Bootstrap starting.
I20260927 10:49:54.929554 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 9294
I20260927 10:49:54.929685 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:49:54.929957 6270 external_mini_cluster.cc:1428] Setting key 6f4b4272fd76612cda226baf50f6a0f4
I20260927 10:49:54.931206 9362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:54.931994 9362 log.cc:824] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:54.934448 9362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb: No bootstrap required, opened a new log
I20260927 10:49:54.939155 9362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e63d4f28c58463082347e3e4bc69aeb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43869 } }
I20260927 10:49:54.939410 9362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:54.939472 9362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e63d4f28c58463082347e3e4bc69aeb, State: Initialized, Role: FOLLOWER
Sep 27 10:49:54 dist-test-slave-c60x krb5kdc[9281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:54.939972 9362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e63d4f28c58463082347e3e4bc69aeb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43869 } }
I20260927 10:49:54.940129 9362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:54.940218 9362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:54.940338 9362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:54.941308 9362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e63d4f28c58463082347e3e4bc69aeb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43869 } }
I20260927 10:49:54.941663 9362 leader_election.cc:302] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e63d4f28c58463082347e3e4bc69aeb; no voters:
I20260927 10:49:54.942050 9362 leader_election.cc:288] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:54.942155 9367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:54.942370 9367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [term 1 LEADER]: Becoming Leader. State: Replica: 5e63d4f28c58463082347e3e4bc69aeb, State: Running, Role: LEADER
I20260927 10:49:54.942741 9367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e63d4f28c58463082347e3e4bc69aeb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43869 } }
I20260927 10:49:54.943500 9362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:54.944479 9368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e63d4f28c58463082347e3e4bc69aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e63d4f28c58463082347e3e4bc69aeb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43869 } } }
I20260927 10:49:54.944664 9368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:54.945056 9369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e63d4f28c58463082347e3e4bc69aeb. Latest consensus state: current_term: 1 leader_uuid: "5e63d4f28c58463082347e3e4bc69aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e63d4f28c58463082347e3e4bc69aeb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43869 } } }
I20260927 10:49:54.945209 9369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:54.946522 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:54.931236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.6.31.190:43869)
0927 10:49:54.931721 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:54.931737 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:54.931774 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:49:54.932632 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:54.932659 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:54.933043 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:54.933330 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:54.934269 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:54.935425 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:54.936665 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:54.937162 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:54.937292 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:54.940512 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:54.940517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:54.940531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:54.940561 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:54.942598 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:54.943123 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:54.943129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:54.943135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:54.943272 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:54.943593 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:54.943596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:54.943598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:54.944116 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:54.944320 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:54.945246 (+ 926us) server_negotiation.cc:299] Negotiation successful
0927 10:49:54.945558 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":151,"threads_started":1}
W20260927 10:49:54.948625 9383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:54.948704 9383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:54.948782 9376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:54.949512 9376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:54.954962 9376 catalog_manager.cc:1347] Generated new cluster ID: 6713b1f81e4646478dd4b9d16c443b83
I20260927 10:49:54.955029 9376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:54.976313 9376 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:54.977603 9376 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:54.993666 9376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e63d4f28c58463082347e3e4bc69aeb: Generated new TSK 0
I20260927 10:49:54.994910 9376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:49:55.034255 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:43869
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:42095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:55.169473 9388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:55.169749 9388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:55.169791 9388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:55.173738 9388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:55.173800 9388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:55.173857 9388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:49:55.178992 9388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:43869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:55.180099 9388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:55.181080 9388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:55.190635 9393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:55.190636 9394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:55.190635 9396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:55.190804 9388 server_base.cc:1034] running on GCE node
I20260927 10:49:55.191398 9388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:55.192049 9388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:55.193241 9388 hybrid_clock.cc:648] HybridClock initialized: now 1790506195193215 us; error 36 us; skew 500 ppm
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506195, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:55.196265 9388 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:49:55.197212 9388 webserver.cc:460] Webserver started at http://127.6.31.129:46313/ using document root <none> and password file <none>
I20260927 10:49:55.197734 9388 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:55.197793 9388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:55.197957 9388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:55.200035 9388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "e5ada1a369b944828cee5608d14485a0"
format_stamp: "Formatted at 2026-09-27 10:49:55 on dist-test-slave-c60x"
server_key: "7591ae7d6a4948e4a2c4ebb94ab3cd0e"
server_key_iv: "e64a3212514891739b1ac1490572d2a0"
server_key_version: "encryptionkey@0"
I20260927 10:49:55.200536 9388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "e5ada1a369b944828cee5608d14485a0"
format_stamp: "Formatted at 2026-09-27 10:49:55 on dist-test-slave-c60x"
server_key: "7591ae7d6a4948e4a2c4ebb94ab3cd0e"
server_key_iv: "e64a3212514891739b1ac1490572d2a0"
server_key_version: "encryptionkey@0"
I20260927 10:49:55.204216 9388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:49:55.206671 9403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:55.207944 9388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:55.208107 9388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "e5ada1a369b944828cee5608d14485a0"
format_stamp: "Formatted at 2026-09-27 10:49:55 on dist-test-slave-c60x"
server_key: "7591ae7d6a4948e4a2c4ebb94ab3cd0e"
server_key_iv: "e64a3212514891739b1ac1490572d2a0"
server_key_version: "encryptionkey@0"
I20260927 10:49:55.208217 9388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:55.240253 9388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:55.243131 9388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:55.243332 9388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:55.243920 9388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:55.244814 9388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:55.244860 9388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:55.244927 9388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:55.244956 9388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:55.253959 9388 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:45877
I20260927 10:49:55.253979 9516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:45877 every 8 connection(s)
I20260927 10:49:55.254819 9388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:49:55.261740 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 9388
I20260927 10:49:55.261862 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:49:55.262145 6270 external_mini_cluster.cc:1428] Setting key 5fbb8457406362ce88eec1936099e724
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506195, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:55.269351 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:55.256827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:60107 (local address 127.6.31.190:43869)
0927 10:49:55.257062 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:55.257067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:55.257761 (+ 694us) server_negotiation.cc:407] Connection header received
0927 10:49:55.259070 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:55.259072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:55.259173 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:55.259247 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:55.260391 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.260910 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.262357 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.262645 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.262756 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:55.265754 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:55.265757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:55.265759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:55.265780 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:55.267282 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:55.267854 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:55.267858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:55.267860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:55.267938 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:55.268269 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:55.268274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:55.268276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:55.268449 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:55.268562 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:55.269053 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 10:49:55.269181 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 10:49:55.270128 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:55.257187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43869 (local address 127.6.31.129:60107)
0927 10:49:55.257616 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:55.257656 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:55.258880 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:55.259403 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:55.259409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:55.259746 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:55.260266 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:55.260278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.261084 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.261086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:55.262242 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:55.262253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.262982 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.262986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:55.263207 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:55.263859 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:55.263875 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:55.265586 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:55.267399 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:55.267405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:55.267418 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:55.267752 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:55.268032 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:55.268036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:55.268038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:55.268176 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:55.268558 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:55.268564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:55.268806 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 10:49:55.269010 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 10:49:55.269317 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:49:55.271442 9517 heartbeater.cc:347] Connected to a master server at 127.6.31.190:43869
I20260927 10:49:55.271697 9517 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:55.272154 9517 heartbeater.cc:511] Master 127.6.31.190:43869 requested a full tablet report, sending...
I20260927 10:49:55.273646 9322 ts_manager.cc:194] Registered new tserver with Master: e5ada1a369b944828cee5608d14485a0 (127.6.31.129:45877)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:49:55.274812 9322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:60107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:49:55.289981 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:43869
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:42095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:55.398138 9522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:55.398373 9522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:55.398414 9522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:55.402297 9522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:55.402352 9522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:55.402410 9522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:49:55.407656 9522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:43869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:55.408658 9522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:55.409622 9522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:55.419286 9530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:55.419337 9528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:55.419476 9522 server_base.cc:1034] running on GCE node
W20260927 10:49:55.419346 9527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:55.419859 9522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:55.420481 9522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:55.421672 9522 hybrid_clock.cc:648] HybridClock initialized: now 1790506195421657 us; error 30 us; skew 500 ppm
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506195, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:55.424813 9522 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:49:55.425774 9522 webserver.cc:460] Webserver started at http://127.6.31.130:41397/ using document root <none> and password file <none>
I20260927 10:49:55.426282 9522 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:55.426352 9522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:55.426514 9522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:55.428565 9522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "50a8caacb52f49b2bd59de36a73784da"
format_stamp: "Formatted at 2026-09-27 10:49:55 on dist-test-slave-c60x"
server_key: "4185a35f596d928cecc68c69ffad20dd"
server_key_iv: "c0b16c52e7c43a9add2b1a135285bbe8"
server_key_version: "encryptionkey@0"
I20260927 10:49:55.429085 9522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "50a8caacb52f49b2bd59de36a73784da"
format_stamp: "Formatted at 2026-09-27 10:49:55 on dist-test-slave-c60x"
server_key: "4185a35f596d928cecc68c69ffad20dd"
server_key_iv: "c0b16c52e7c43a9add2b1a135285bbe8"
server_key_version: "encryptionkey@0"
I20260927 10:49:55.432607 9522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:49:55.435036 9537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:55.436311 9522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:55.436450 9522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "50a8caacb52f49b2bd59de36a73784da"
format_stamp: "Formatted at 2026-09-27 10:49:55 on dist-test-slave-c60x"
server_key: "4185a35f596d928cecc68c69ffad20dd"
server_key_iv: "c0b16c52e7c43a9add2b1a135285bbe8"
server_key_version: "encryptionkey@0"
I20260927 10:49:55.436560 9522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:55.452345 9522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:55.455227 9522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:55.455433 9522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:55.456053 9522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:55.456858 9522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:55.456929 9522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:55.457000 9522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:55.457053 9522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:55.465281 9522 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:41829
I20260927 10:49:55.465349 9650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:41829 every 8 connection(s)
I20260927 10:49:55.466241 9522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506195, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:55.476075 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 9522
I20260927 10:49:55.476187 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:49:55.476554 6270 external_mini_cluster.cc:1428] Setting key 6baf89757347b8a6c6eca643d5870af7
I20260927 10:49:55.480979 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:55.468175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:43539 (local address 127.6.31.190:43869)
0927 10:49:55.468349 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:55.468363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:55.469142 (+ 779us) server_negotiation.cc:407] Connection header received
0927 10:49:55.470442 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:55.470445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:55.470493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:55.470579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:55.471764 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.472301 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.473418 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.473716 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.473803 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:55.476577 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:55.476581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:55.476584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:55.476615 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:55.478359 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:55.479015 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:55.479020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:55.479022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:55.479155 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:55.479625 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:55.479632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:55.479637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:55.479898 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:55.479977 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:55.480733 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 10:49:55.480857 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 10:49:55.482244 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:55.468559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43869 (local address 127.6.31.130:43539)
0927 10:49:55.468990 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:55.469030 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:55.470239 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:55.470734 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:55.470740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:55.471144 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:55.471618 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:55.471631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.472411 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.472413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:55.473289 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:55.473298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.473845 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.473848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:55.473978 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:55.474581 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:55.474598 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:55.476332 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:55.478501 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:55.478507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:55.478522 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:55.478879 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:55.479275 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:55.479279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:55.479281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:55.479488 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:55.480064 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:55.480073 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:55.480424 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 10:49:55.481173 (+ 749us) client_negotiation.cc:240] Negotiation successful
0927 10:49:55.481497 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":122,"threads_started":1}
I20260927 10:49:55.483525 9651 heartbeater.cc:347] Connected to a master server at 127.6.31.190:43869
I20260927 10:49:55.483800 9651 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:55.484412 9651 heartbeater.cc:511] Master 127.6.31.190:43869 requested a full tablet report, sending...
I20260927 10:49:55.485551 9322 ts_manager.cc:194] Registered new tserver with Master: 50a8caacb52f49b2bd59de36a73784da (127.6.31.130:41829)
I20260927 10:49:55.486253 9322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:43539
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:49:55.506107 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:43869
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:42095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:55.617079 9656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:55.617360 9656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:55.617441 9656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:55.621421 9656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:55.621511 9656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:55.621608 9656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:49:55.627072 9656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:43869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:55.628189 9656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:55.629204 9656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:55.638788 9662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:55.638835 9661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:55.638788 9664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:55.639140 9656 server_base.cc:1034] running on GCE node
I20260927 10:49:55.639658 9656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:55.640357 9656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:55.641532 9656 hybrid_clock.cc:648] HybridClock initialized: now 1790506195641510 us; error 31 us; skew 500 ppm
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506195, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:55.644444 9656 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:49:55.645393 9656 webserver.cc:460] Webserver started at http://127.6.31.131:46799/ using document root <none> and password file <none>
I20260927 10:49:55.645958 9656 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:55.646049 9656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:55.646279 9656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:55.648375 9656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "c120b94b11ad430fa2a739c984a258e6"
format_stamp: "Formatted at 2026-09-27 10:49:55 on dist-test-slave-c60x"
server_key: "b654dde6fdc6d147082dc6c179f6d0e6"
server_key_iv: "072b564337332f217d7bf2ba2defd20a"
server_key_version: "encryptionkey@0"
I20260927 10:49:55.648936 9656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "c120b94b11ad430fa2a739c984a258e6"
format_stamp: "Formatted at 2026-09-27 10:49:55 on dist-test-slave-c60x"
server_key: "b654dde6fdc6d147082dc6c179f6d0e6"
server_key_iv: "072b564337332f217d7bf2ba2defd20a"
server_key_version: "encryptionkey@0"
I20260927 10:49:55.652729 9656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 10:49:55.655189 9671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:55.656358 9656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:55.656492 9656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "c120b94b11ad430fa2a739c984a258e6"
format_stamp: "Formatted at 2026-09-27 10:49:55 on dist-test-slave-c60x"
server_key: "b654dde6fdc6d147082dc6c179f6d0e6"
server_key_iv: "072b564337332f217d7bf2ba2defd20a"
server_key_version: "encryptionkey@0"
I20260927 10:49:55.656610 9656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:55.668313 9656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:55.671394 9656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:55.671594 9656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:55.672168 9656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:55.672987 9656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:55.673060 9656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:55.673132 9656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:55.673185 9656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:55.681936 9656 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:33401
I20260927 10:49:55.682053 9784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:33401 every 8 connection(s)
I20260927 10:49:55.682858 9656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506195, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:55.692373 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 9656
I20260927 10:49:55.692490 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:49:55.692780 6270 external_mini_cluster.cc:1428] Setting key 9c7ef7ccd7ecfb6d2207eceb53dcfacc
I20260927 10:49:55.696514 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:55.684663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:36623 (local address 127.6.31.190:43869)
0927 10:49:55.684793 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:55.684804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:55.685610 (+ 806us) server_negotiation.cc:407] Connection header received
0927 10:49:55.686962 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:55.686966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:55.687035 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:55.687128 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:55.688246 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.688782 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.689891 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.690138 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.690248 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:55.692680 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:55.692683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:55.692686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:55.692718 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:55.694379 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:55.694910 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:55.694913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:55.694915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:55.695029 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:55.695461 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:55.695465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:55.695466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:55.695655 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:55.695712 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:55.696300 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 10:49:55.696399 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:49:55.697386 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:55.684946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43869 (local address 127.6.31.131:36623)
0927 10:49:55.685399 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:55.685505 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:55.686765 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:55.687317 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:55.687323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:55.687667 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:55.688111 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:55.688123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.688903 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.688906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:55.689769 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:55.689779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.690283 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.690286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:55.690449 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:55.690976 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:55.690994 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:55.692504 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:55.694486 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:55.694491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:55.694506 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:55.694788 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:55.695185 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:55.695188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:55.695190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:55.695345 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:55.695777 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:55.695782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:55.696068 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 10:49:55.696262 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 10:49:55.696526 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":141,"threads_started":1}
I20260927 10:49:55.698518 9785 heartbeater.cc:347] Connected to a master server at 127.6.31.190:43869
I20260927 10:49:55.698781 9785 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:55.699371 9785 heartbeater.cc:511] Master 127.6.31.190:43869 requested a full tablet report, sending...
I20260927 10:49:55.700475 9322 ts_manager.cc:194] Registered new tserver with Master: c120b94b11ad430fa2a739c984a258e6 (127.6.31.131:33401)
I20260927 10:49:55.701035 9322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:36623
I20260927 10:49:55.706516 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506195, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506195, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:55.731766 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:55.723199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.6.31.190:43869)
0927 10:49:55.723363 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:55.723367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:55.723522 (+ 155us) server_negotiation.cc:407] Connection header received
0927 10:49:55.723696 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:55.723700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:55.723751 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:55.723839 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:55.724541 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.725071 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.725904 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:55.726128 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:55.726237 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:55.728700 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:55.728703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:55.728705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:55.728730 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:55.730011 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:55.730433 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:55.730438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:55.730439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:55.730519 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:55.730797 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:55.730802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:55.730803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:55.731054 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:55.731232 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:55.731468 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 10:49:55.731597 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:49:55.732155 9322 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33220
I20260927 10:49:55.732992 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 9388
I20260927 10:49:55.738695 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 9522
I20260927 10:49:55.744310 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 9656
I20260927 10:49:55.750247 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 9294
2026-09-27T10:49:55Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2186 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9797](info): set up 2 sockets
Sep 27 10:49:55 dist-test-slave-c60x krb5kdc[9797](info): commencing operation
krb5kdc: starting...
W20260927 10:49:56.795185 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:49:56 dist-test-slave-c60x krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:49:56.917901 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:44377
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:57.026160 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:57.026404 9810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:57.026444 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:57.030373 9810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:57.030431 9810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:57.030450 9810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:57.030467 9810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:57.030483 9810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:57.035629 9810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:44377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:57.036589 9810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:57.037525 9810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:57.046006 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:57.046105 9810 server_base.cc:1034] running on GCE node
W20260927 10:49:57.046022 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:57.045994 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:57.046814 9810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:57.047946 9810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:57.049134 9810 hybrid_clock.cc:648] HybridClock initialized: now 1790506197049102 us; error 43 us; skew 500 ppm
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:57.051765 9810 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:49:57.052636 9810 webserver.cc:460] Webserver started at http://127.6.31.190:40669/ using document root <none> and password file <none>
I20260927 10:49:57.053233 9810 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:57.053321 9810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:57.053542 9810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:57.055593 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "2f2f6cd6d56ce26b9cc7951f5446f033"
server_key_iv: "c117741e361aff9bdae346943435a46a"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.056152 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "2f2f6cd6d56ce26b9cc7951f5446f033"
server_key_iv: "c117741e361aff9bdae346943435a46a"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.060111 9810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:49:57.062563 9825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.063871 9810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:57.064013 9810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "2f2f6cd6d56ce26b9cc7951f5446f033"
server_key_iv: "c117741e361aff9bdae346943435a46a"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.064128 9810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:57.095265 9810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:57.098469 9810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:57.098757 9810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:57.105613 9810 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:44377
I20260927 10:49:57.105643 9877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:44377 every 8 connection(s)
I20260927 10:49:57.106519 9810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:57.109148 9878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:57.114163 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 9810
I20260927 10:49:57.114259 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:49:57.114341 9878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224: Bootstrap starting.
I20260927 10:49:57.114574 6270 external_mini_cluster.cc:1428] Setting key 050546fcff46c841b6edbf357e6cda19
I20260927 10:49:57.116765 9878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:57.117473 9878 log.cc:824] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:57.119457 9878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224: No bootstrap required, opened a new log
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:57.123879 9878 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44377 } }
I20260927 10:49:57.124109 9878 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:57.124179 9878 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a0bc2fb99d4ad488fd4ac58d4eb224, State: Initialized, Role: FOLLOWER
I20260927 10:49:57.124619 9878 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44377 } }
I20260927 10:49:57.124786 9878 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:57.124871 9878 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:57.124989 9878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:57.125934 9878 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44377 } }
I20260927 10:49:57.126289 9878 leader_election.cc:302] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5a0bc2fb99d4ad488fd4ac58d4eb224; no voters:
I20260927 10:49:57.126637 9878 leader_election.cc:288] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:57.126755 9883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:57.126964 9883 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [term 1 LEADER]: Becoming Leader. State: Replica: b5a0bc2fb99d4ad488fd4ac58d4eb224, State: Running, Role: LEADER
I20260927 10:49:57.127336 9883 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44377 } }
I20260927 10:49:57.127954 9878 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:57.129007 9884 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a0bc2fb99d4ad488fd4ac58d4eb224. Latest consensus state: current_term: 1 leader_uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44377 } } }
I20260927 10:49:57.129320 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:57.116000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.6.31.190:44377)
0927 10:49:57.116409 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:57.116427 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:57.116468 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:49:57.117139 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:57.117156 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:57.117471 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:57.117776 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:57.118612 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.119711 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.120818 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.121253 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.121362 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:57.124205 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:57.124210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:57.124222 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:57.124248 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:57.126154 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.126664 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.126671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.126678 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.126827 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.127129 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.127132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.127134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.127606 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:57.127789 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:57.128047 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 10:49:57.128348 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
I20260927 10:49:57.129366 9884 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:57.129251 9885 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a0bc2fb99d4ad488fd4ac58d4eb224" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44377 } } }
I20260927 10:49:57.129429 9885 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:57.129721 9892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:57.132723 9892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:57.137951 9892 catalog_manager.cc:1347] Generated new cluster ID: 009822c630cf436a88ad488c40b8dc2e
I20260927 10:49:57.138015 9892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:57.150134 9892 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:57.151036 9892 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:57.163456 9892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5a0bc2fb99d4ad488fd4ac58d4eb224: Generated new TSK 0
I20260927 10:49:57.164269 9892 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:49:57.198298 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44377
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:57.307897 9904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:57.308149 9904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:57.308223 9904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:57.312232 9904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:57.312295 9904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:57.312355 9904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:49:57.317660 9904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:57.318737 9904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:57.319761 9904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:57.329141 9910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:57.329161 9912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:57.329161 9909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:57.329396 9904 server_base.cc:1034] running on GCE node
I20260927 10:49:57.329936 9904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:57.330600 9904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:57.331756 9904 hybrid_clock.cc:648] HybridClock initialized: now 1790506197331724 us; error 42 us; skew 500 ppm
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:57.334493 9904 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:49:57.335386 9904 webserver.cc:460] Webserver started at http://127.6.31.129:33151/ using document root <none> and password file <none>
I20260927 10:49:57.336042 9904 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:57.336171 9904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:57.336431 9904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:57.338407 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "73cf7839f4824733a6bf7b523d1cd15f"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "245805b4fe1a4586315c0cd1585864be"
server_key_iv: "37d40c6819febe8b80991ea5e5290f51"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.338979 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "73cf7839f4824733a6bf7b523d1cd15f"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "245805b4fe1a4586315c0cd1585864be"
server_key_iv: "37d40c6819febe8b80991ea5e5290f51"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.342478 9904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:49:57.344787 9919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.345948 9904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:49:57.346086 9904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "73cf7839f4824733a6bf7b523d1cd15f"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "245805b4fe1a4586315c0cd1585864be"
server_key_iv: "37d40c6819febe8b80991ea5e5290f51"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.346194 9904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:57.369692 9904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:57.372816 9904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:57.373061 9904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:57.373684 9904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:57.374600 9904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:57.374675 9904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.374742 9904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:57.374786 9904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.384475 9904 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:44153
I20260927 10:49:57.384480 10032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:44153 every 8 connection(s)
I20260927 10:49:57.385360 9904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:57.394807 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 9904
I20260927 10:49:57.394922 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:49:57.395282 6270 external_mini_cluster.cc:1428] Setting key 0e722f9ed4306fac1b7626fb72724e94
I20260927 10:49:57.399128 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:57.387175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:50911 (local address 127.6.31.190:44377)
0927 10:49:57.387312 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:57.387316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:57.388098 (+ 782us) server_negotiation.cc:407] Connection header received
0927 10:49:57.389389 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:57.389392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:57.389437 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:57.389515 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:57.390642 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.391305 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.392373 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.392601 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.392687 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:57.395249 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:57.395252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:57.395254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:57.395277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:57.397079 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.397584 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.397588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.397590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.397676 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.398011 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.398016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.398018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.398198 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:57.398300 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:57.398795 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 10:49:57.398926 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:49:57.399947 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:57.387533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44377 (local address 127.6.31.129:50911)
0927 10:49:57.387950 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:57.387990 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:57.389191 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:57.389665 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:57.389672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:57.390010 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:57.390468 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:57.390480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.391421 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.391423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:57.392260 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:57.392269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.392730 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.392732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:57.392865 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:57.393473 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:57.393489 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:57.395042 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:57.397191 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:57.397196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:57.397210 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:57.397477 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:57.397767 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:57.397771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:57.397773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:57.397916 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:57.398311 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:57.398319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:57.398565 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 10:49:57.398755 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 10:49:57.399058 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260927 10:49:57.401180 10033 heartbeater.cc:347] Connected to a master server at 127.6.31.190:44377
I20260927 10:49:57.401435 10033 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:57.402034 10033 heartbeater.cc:511] Master 127.6.31.190:44377 requested a full tablet report, sending...
I20260927 10:49:57.403764 9842 ts_manager.cc:194] Registered new tserver with Master: 73cf7839f4824733a6bf7b523d1cd15f (127.6.31.129:44153)
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:49:57.405316 9842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:50911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:49:57.421990 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44377
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:57.531256 10038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:57.531477 10038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:57.531518 10038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:57.535487 10038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:57.535537 10038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:57.535640 10038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:49:57.540661 10038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:57.541519 10038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:57.542408 10038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:57.551783 10046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:57.551766 10044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:57.551771 10043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:57.552016 10038 server_base.cc:1034] running on GCE node
I20260927 10:49:57.552443 10038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:57.553035 10038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:57.554227 10038 hybrid_clock.cc:648] HybridClock initialized: now 1790506197554219 us; error 41 us; skew 500 ppm
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:57.556916 10038 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:49:57.557770 10038 webserver.cc:460] Webserver started at http://127.6.31.130:43643/ using document root <none> and password file <none>
I20260927 10:49:57.558277 10038 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:57.558332 10038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:57.558488 10038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:57.560483 10038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "a32036ab696d4d9a8278254c1cc1065e"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "54cfe07d90d9e0a748b533f539437da2"
server_key_iv: "f268f1ad024312ebd8853dc1938c6472"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.560986 10038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "a32036ab696d4d9a8278254c1cc1065e"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "54cfe07d90d9e0a748b533f539437da2"
server_key_iv: "f268f1ad024312ebd8853dc1938c6472"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.564530 10038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:49:57.566813 10053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.568063 10038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:57.568212 10038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "a32036ab696d4d9a8278254c1cc1065e"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "54cfe07d90d9e0a748b533f539437da2"
server_key_iv: "f268f1ad024312ebd8853dc1938c6472"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.568329 10038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:57.582407 10038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:57.585367 10038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:57.585569 10038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:57.586167 10038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:57.586949 10038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:57.587023 10038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.587159 10038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:57.587204 10038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.595973 10038 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:44891
I20260927 10:49:57.595990 10166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:44891 every 8 connection(s)
I20260927 10:49:57.596815 10038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:49:57.598274 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 10038
I20260927 10:49:57.598410 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:49:57.598728 6270 external_mini_cluster.cc:1428] Setting key 7ee5ca57baf3ca8d629f19df13695788
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:49:57.610459 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:57.598526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:41787 (local address 127.6.31.190:44377)
0927 10:49:57.599201 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:57.599205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:57.599637 (+ 432us) server_negotiation.cc:407] Connection header received
0927 10:49:57.601096 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:57.601101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:57.601162 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:57.601247 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:57.602291 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.603119 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.604197 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.604604 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.604724 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:57.607029 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:57.607034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:57.607038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:57.607070 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:57.608656 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.609151 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.609154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.609155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.609226 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.609507 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.609509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.609510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.609668 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:57.609785 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:57.610238 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 10:49:57.610329 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
I20260927 10:49:57.611443 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:57.598960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44377 (local address 127.6.31.130:41787)
0927 10:49:57.599490 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:57.599541 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:57.600820 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:57.601378 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:57.601384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:57.601705 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:57.602151 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:57.602164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.603229 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.603232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:57.604056 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:57.604064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.604748 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.604751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:57.604886 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:57.605489 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:57.605505 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:57.606866 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:57.608798 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:57.608802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:57.608812 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:57.609058 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:57.609306 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:57.609308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:57.609310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:57.609428 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:57.609793 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:57.609801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:57.610035 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 10:49:57.610185 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 10:49:57.610410 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":184,"threads_started":1}
I20260927 10:49:57.612672 10167 heartbeater.cc:347] Connected to a master server at 127.6.31.190:44377
I20260927 10:49:57.612901 10167 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:57.613368 10167 heartbeater.cc:511] Master 127.6.31.190:44377 requested a full tablet report, sending...
I20260927 10:49:57.614470 9842 ts_manager.cc:194] Registered new tserver with Master: a32036ab696d4d9a8278254c1cc1065e (127.6.31.130:44891)
I20260927 10:49:57.615331 9842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:41787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:49:57.628865 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44377
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:57.739712 10172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:57.739974 10172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:57.740020 10172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:57.744063 10172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:57.744118 10172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:57.744235 10172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:49:57.749693 10172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:57.750789 10172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:57.751811 10172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:57.761160 10178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:57.761206 10177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:57.761157 10180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:57.761487 10172 server_base.cc:1034] running on GCE node
I20260927 10:49:57.761874 10172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:57.762481 10172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:57.763660 10172 hybrid_clock.cc:648] HybridClock initialized: now 1790506197763644 us; error 33 us; skew 500 ppm
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:57.766337 10172 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:49:57.767249 10172 webserver.cc:460] Webserver started at http://127.6.31.131:34823/ using document root <none> and password file <none>
I20260927 10:49:57.767799 10172 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:57.767884 10172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:57.768106 10172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:57.770032 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "12ef3a2580514273ae83255f9fdf1acd"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "9a6194dbe02872f91dbf9e2bbfd52864"
server_key_iv: "5b76d96e5c7997a4c63a72a6d4f9c13c"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.770531 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "12ef3a2580514273ae83255f9fdf1acd"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "9a6194dbe02872f91dbf9e2bbfd52864"
server_key_iv: "5b76d96e5c7997a4c63a72a6d4f9c13c"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.774228 10172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:49:57.776625 10187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.777930 10172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:57.778061 10172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "12ef3a2580514273ae83255f9fdf1acd"
format_stamp: "Formatted at 2026-09-27 10:49:57 on dist-test-slave-c60x"
server_key: "9a6194dbe02872f91dbf9e2bbfd52864"
server_key_iv: "5b76d96e5c7997a4c63a72a6d4f9c13c"
server_key_version: "encryptionkey@0"
I20260927 10:49:57.778172 10172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:57.792320 10172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:57.795679 10172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:57.795881 10172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:57.796461 10172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:57.797238 10172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:57.797281 10172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.797345 10172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:57.797392 10172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:57.806203 10172 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:34381
I20260927 10:49:57.806242 10300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:34381 every 8 connection(s)
I20260927 10:49:57.807174 10172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:49:57.815464 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 10172
I20260927 10:49:57.815611 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:49:57.815896 6270 external_mini_cluster.cc:1428] Setting key b04bbef1ca0258d33795b40195ff024e
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:57.820703 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:57.808990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:60381 (local address 127.6.31.190:44377)
0927 10:49:57.809138 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:57.809141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:57.809821 (+ 680us) server_negotiation.cc:407] Connection header received
0927 10:49:57.811137 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:57.811141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:57.811200 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:57.811275 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:57.812449 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.813009 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.814113 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.814335 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.814425 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:57.817286 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:57.817289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:57.817292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:57.817326 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:57.818764 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.819330 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.819333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.819334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.819425 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.819728 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.819730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.819731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.819900 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:57.819986 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:57.820492 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 10:49:57.820587 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:49:57.821488 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:57.809229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44377 (local address 127.6.31.131:60381)
0927 10:49:57.809669 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:57.809710 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:57.810903 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:57.811465 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:57.811472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:57.811807 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:57.812275 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:57.812287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.813121 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.813124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:57.813984 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:57.813992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.814501 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.814505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:57.814655 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:57.815332 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:57.815349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:57.817134 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:57.818925 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:57.818930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:57.818942 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:57.819212 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:57.819513 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:57.819515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:57.819516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:57.819649 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:57.820027 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:57.820032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:57.820254 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 10:49:57.820412 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 10:49:57.820656 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260927 10:49:57.822584 10301 heartbeater.cc:347] Connected to a master server at 127.6.31.190:44377
I20260927 10:49:57.822834 10301 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:57.823441 10301 heartbeater.cc:511] Master 127.6.31.190:44377 requested a full tablet report, sending...
I20260927 10:49:57.824445 9842 ts_manager.cc:194] Registered new tserver with Master: 12ef3a2580514273ae83255f9fdf1acd (127.6.31.131:34381)
I20260927 10:49:57.825018 9842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:60381
I20260927 10:49:57.829900 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:49:57.855348 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:57.845746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.6.31.129:44153)
0927 10:49:57.846032 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:57.846048 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:57.846075 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:49:57.846206 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:57.846219 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:57.846408 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:57.846557 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:57.847285 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.848134 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.848949 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.849361 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.849480 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:57.852002 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:57.852020 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:57.852033 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:57.852055 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:57.853505 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.853970 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.853976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.853978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.854065 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.854337 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.854342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.854344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.854592 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:57.854799 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:57.854979 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 10:49:57.855156 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:57.865852 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:57.858131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.6.31.190:44377)
0927 10:49:57.858283 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:57.858287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:57.858300 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:49:57.858468 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:57.858471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:57.858516 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:57.858591 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:57.859230 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.859760 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.860508 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:57.860715 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:57.860794 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:57.862954 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:57.862957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:57.862959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:57.862982 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:57.864454 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.864797 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.864799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.864801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.864869 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:57.865082 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:57.865084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:57.865085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:57.865285 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:57.865384 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:57.865551 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 10:49:57.865638 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 10:49:57.866281 9842 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58180
I20260927 10:49:57.866878 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 9904
I20260927 10:49:57.873404 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 10038
I20260927 10:49:57.878677 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 10172
I20260927 10:49:57.883756 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 9810
2026-09-27T10:49:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2133 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[10313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[10313](info): set up 2 sockets
Sep 27 10:49:57 dist-test-slave-c60x krb5kdc[10313](info): commencing operation
krb5kdc: starting...
W20260927 10:49:58.927803 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:49:59 dist-test-slave-c60x krb5kdc[10313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:49:59.047851 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38351
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:38351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:59.157744 10326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:59.158017 10326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:59.158061 10326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:59.162005 10326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:59.162062 10326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:59.162082 10326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:59.162098 10326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:59.162114 10326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:59.167289 10326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:38351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:59.168282 10326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:59.169232 10326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:59.177700 10331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:59.177692 10332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:59.177793 10326 server_base.cc:1034] running on GCE node
W20260927 10:49:59.177692 10334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:59.178529 10326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:59.179747 10326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:59.180939 10326 hybrid_clock.cc:648] HybridClock initialized: now 1790506199180865 us; error 89 us; skew 500 ppm
Sep 27 10:49:59 dist-test-slave-c60x krb5kdc[10313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:59.192359 10326 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:49:59.193231 10326 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:59.193338 10326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:59.193550 10326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:59.195794 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "b96abaabbcceeef0eda0d11d1606877c"
server_key_iv: "56e4c1502ea0513ec06fd038de4df3e9"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.196415 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "b96abaabbcceeef0eda0d11d1606877c"
server_key_iv: "56e4c1502ea0513ec06fd038de4df3e9"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.200265 10326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:49:59.202790 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.204245 10326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:49:59.204396 10326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "b96abaabbcceeef0eda0d11d1606877c"
server_key_iv: "56e4c1502ea0513ec06fd038de4df3e9"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.204514 10326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:59.223665 10326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:59.227536 10326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:59.227731 10326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:59.235062 10326 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:38351
I20260927 10:49:59.235390 10392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:38351 every 8 connection(s)
I20260927 10:49:59.235780 10326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:59.238518 10393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:59.243882 10393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9: Bootstrap starting.
I20260927 10:49:59.244292 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 10326
I20260927 10:49:59.244391 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:49:59.244663 6270 external_mini_cluster.cc:1428] Setting key 9340908196e4c4dac78afb373c2cad56
I20260927 10:49:59.246457 10393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:59.247339 10393 log.cc:824] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:59.249161 10393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9: No bootstrap required, opened a new log
I20260927 10:49:59.252600 10393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38351 } }
I20260927 10:49:59.252833 10393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:59.252910 10393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7c1fbd9c0e4db4b9398281f81c02a9, State: Initialized, Role: FOLLOWER
I20260927 10:49:59.253343 10393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38351 } }
Sep 27 10:49:59 dist-test-slave-c60x krb5kdc[10313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:59.253491 10393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:59.253579 10393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:59.253686 10393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:59.254668 10393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38351 } }
I20260927 10:49:59.255054 10393 leader_election.cc:302] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d7c1fbd9c0e4db4b9398281f81c02a9; no voters:
I20260927 10:49:59.255417 10393 leader_election.cc:288] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:59.255534 10398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:59.255728 10398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [term 1 LEADER]: Becoming Leader. State: Replica: 4d7c1fbd9c0e4db4b9398281f81c02a9, State: Running, Role: LEADER
I20260927 10:49:59.256105 10398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38351 } }
I20260927 10:49:59.256772 10393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:59.257292 10398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d7c1fbd9c0e4db4b9398281f81c02a9. Latest consensus state: current_term: 1 leader_uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38351 } } }
I20260927 10:49:59.258025 10398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:59.257922 10399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7c1fbd9c0e4db4b9398281f81c02a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38351 } } }
I20260927 10:49:59.258314 10399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:59.259383 10407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:59.260272 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:59.245997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.6.31.190:38351)
0927 10:49:59.246416 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:59.246437 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:59.246479 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:49:59.247361 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:59.247383 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:59.247786 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:59.248103 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:59.248882 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.249881 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.250727 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.251268 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.251443 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:59.254073 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:59.254079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:59.254100 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:59.254127 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:59.256277 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.256780 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.256785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.256790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.256921 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.257259 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.257263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.257265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.257816 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:59.258669 (+ 853us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:59.258824 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 10:49:59.259205 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
I20260927 10:49:59.262436 10407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:59.268249 10407 catalog_manager.cc:1347] Generated new cluster ID: 45e0e1c2dae741a2b9b8148738e58770
I20260927 10:49:59.268321 10407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:59.287791 10407 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:59.289069 10407 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:59.301385 10407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9: Generated new TSK 0
I20260927 10:49:59.302228 10407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:49:59.333854 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38351
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:59.442404 10419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:59.442731 10419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:59.442811 10419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:59.446985 10419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:59.447093 10419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:59.447211 10419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:49:59.452611 10419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:59.453804 10419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:59.454859 10419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:59.464553 10427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:59.464553 10425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:59.464587 10424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:59.464632 10419 server_base.cc:1034] running on GCE node
I20260927 10:49:59.465271 10419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:59.465926 10419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:59.467126 10419 hybrid_clock.cc:648] HybridClock initialized: now 1790506199467063 us; error 59 us; skew 500 ppm
Sep 27 10:49:59 dist-test-slave-c60x krb5kdc[10313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:59.469951 10419 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:49:59.470746 10419 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:59.470808 10419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:59.471017 10419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:59.473067 10419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "b8242a44cb674199b402f82755d24262"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "71eac1da00c2eb7841b77f49e26bbc6c"
server_key_iv: "bb93110429a8607d0f3155e3cfa8de23"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.473625 10419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "b8242a44cb674199b402f82755d24262"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "71eac1da00c2eb7841b77f49e26bbc6c"
server_key_iv: "bb93110429a8607d0f3155e3cfa8de23"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.477232 10419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:49:59.479871 10433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.481230 10419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:49:59.481395 10419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "b8242a44cb674199b402f82755d24262"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "71eac1da00c2eb7841b77f49e26bbc6c"
server_key_iv: "bb93110429a8607d0f3155e3cfa8de23"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.481516 10419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:59.507440 10419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:59.510952 10419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:59.511174 10419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:59.511842 10419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:59.512629 10419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:59.512676 10419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.512715 10419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:59.512732 10419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.521425 10419 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:42523
I20260927 10:49:59.521499 10546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:42523 every 8 connection(s)
I20260927 10:49:59.522050 10419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:49:59.530380 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 10419
I20260927 10:49:59.530489 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:49:59.530828 6270 external_mini_cluster.cc:1428] Setting key 5bc0ebf02ae8c1526b9d5563c8419646
Sep 27 10:49:59 dist-test-slave-c60x krb5kdc[10313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:59.535818 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:59.523852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:40883 (local address 127.6.31.190:38351)
0927 10:49:59.524000 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:59.524004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:59.524832 (+ 828us) server_negotiation.cc:407] Connection header received
0927 10:49:59.526089 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:59.526092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:59.526140 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:59.526216 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:59.527382 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.527912 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.529039 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.529274 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.529352 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:59.531911 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:59.531914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:59.531917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:59.531939 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:59.533434 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.533982 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.533987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.533991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.534096 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.534480 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.534485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.534489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.534731 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:59.534838 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:59.535432 (+ 594us) server_negotiation.cc:299] Negotiation successful
0927 10:49:59.535575 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:49:59.536563 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:59.524248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38351 (local address 127.6.31.129:40883)
0927 10:49:59.524680 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:59.524722 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:59.525889 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:59.526371 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:59.526377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:59.526722 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:59.527252 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:59.527266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.528039 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.528041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:59.528933 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:59.528943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.529399 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.529402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:59.529541 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:59.530078 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:59.530094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:59.531767 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:59.533553 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:59.533559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:59.533578 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:59.533884 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:59.534223 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:59.534226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:59.534227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:59.534365 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:59.534849 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:59.534855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:59.535196 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 10:49:59.535359 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 10:49:59.535612 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260927 10:49:59.537765 10547 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38351
I20260927 10:49:59.538033 10547 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:59.538494 10547 heartbeater.cc:511] Master 127.6.31.190:38351 requested a full tablet report, sending...
I20260927 10:49:59.540035 10356 ts_manager.cc:194] Registered new tserver with Master: b8242a44cb674199b402f82755d24262 (127.6.31.129:42523)
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:49:59.541275 10356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:40883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:49:59.558764 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38351
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:59.671442 10552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:59.671725 10552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:59.671772 10552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:59.676030 10552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:59.676098 10552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:59.676162 10552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:49:59.681775 10552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:59.682937 10552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:59.684075 10552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:59.693931 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:59.693941 10557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:59.693948 10558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:59.693976 10552 server_base.cc:1034] running on GCE node
I20260927 10:49:59.694664 10552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:59.695392 10552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:59.696590 10552 hybrid_clock.cc:648] HybridClock initialized: now 1790506199696549 us; error 28 us; skew 500 ppm
Sep 27 10:49:59 dist-test-slave-c60x krb5kdc[10313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:59.699466 10552 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:49:59.700193 10552 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:59.700260 10552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:59.700424 10552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:59.702472 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "c305edd9a48d43329a47daae3c7547d9"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "9de3b20daf62065afb749e7b409517cb"
server_key_iv: "d6a4d578cb85c47660012d658b0e2fea"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.703011 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "c305edd9a48d43329a47daae3c7547d9"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "9de3b20daf62065afb749e7b409517cb"
server_key_iv: "d6a4d578cb85c47660012d658b0e2fea"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.706782 10552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:49:59.709203 10566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.710379 10552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:59.710474 10552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "c305edd9a48d43329a47daae3c7547d9"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "9de3b20daf62065afb749e7b409517cb"
server_key_iv: "d6a4d578cb85c47660012d658b0e2fea"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.710572 10552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:59.729261 10552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:59.732610 10552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:59.732836 10552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:59.733424 10552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:59.734264 10552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:59.734310 10552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.734376 10552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:59.734414 10552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.742741 10552 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:35789
I20260927 10:49:59.742810 10679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:35789 every 8 connection(s)
I20260927 10:49:59.743379 10552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:49:59.745100 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 10552
I20260927 10:49:59.745209 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:49:59.745473 6270 external_mini_cluster.cc:1428] Setting key b7c9982785482c70d15eb4516abf3de1
Sep 27 10:49:59 dist-test-slave-c60x krb5kdc[10313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
I20260927 10:49:59.758612 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:59.745087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:58215 (local address 127.6.31.190:38351)
0927 10:49:59.745249 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:59.745254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:59.746207 (+ 953us) server_negotiation.cc:407] Connection header received
0927 10:49:59.747803 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:59.747806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:59.747850 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:59.747924 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:59.749398 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.749924 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.751734 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.751949 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.752036 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:59.755041 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:59.755044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:59.755046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:59.755069 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:59.756552 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.757092 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.757095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.757096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.757170 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.757456 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.757458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.757459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.757644 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:59.757719 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:59.758285 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 10:49:59.758385 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:49:59.759574 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:59.745515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38351 (local address 127.6.31.130:58215)
0927 10:49:59.746027 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:59.746082 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:59.747577 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:59.748102 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:59.748114 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:59.748645 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:59.749253 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:59.749268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.750053 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.750059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:59.751591 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:59.751604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.752111 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.752118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:59.752363 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:59.752998 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:59.753027 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:59.754849 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:59.756659 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:59.756663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:59.756677 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:59.757014 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:59.757251 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:59.757253 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:59.757254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:59.757377 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:59.757742 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:59.757752 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:59.758064 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 10:49:59.758231 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 10:49:59.758605 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260927 10:49:59.760811 10680 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38351
I20260927 10:49:59.761051 10680 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:59.761488 10680 heartbeater.cc:511] Master 127.6.31.190:38351 requested a full tablet report, sending...
I20260927 10:49:59.762506 10356 ts_manager.cc:194] Registered new tserver with Master: c305edd9a48d43329a47daae3c7547d9 (127.6.31.130:35789)
I20260927 10:49:59.763156 10356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:58215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:49:59.774662 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38351
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:59.886448 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:59.886716 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:59.886762 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:59.890877 10685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:59.890925 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:59.890987 10685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:49:59.896145 10685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:59.897064 10685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:59.897986 10685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:59.907454 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:59.907552 10685 server_base.cc:1034] running on GCE node
W20260927 10:49:59.907510 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:59.907718 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:59.908142 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:59.908751 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:59.909916 10685 hybrid_clock.cc:648] HybridClock initialized: now 1790506199909866 us; error 69 us; skew 500 ppm
Sep 27 10:49:59 dist-test-slave-c60x krb5kdc[10313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:59.912787 10685 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:49:59.913540 10685 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:59.913604 10685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:59.913820 10685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:59.915968 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "5a0dec9e4aba4961b1f7b614c2e7439c"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "84152aa89b85877226adad5d21ba88bd"
server_key_iv: "aa9a6af0f162cc5c2092796fef5c09e0"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.916518 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "5a0dec9e4aba4961b1f7b614c2e7439c"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "84152aa89b85877226adad5d21ba88bd"
server_key_iv: "aa9a6af0f162cc5c2092796fef5c09e0"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.920137 10685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:49:59.922559 10699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.923879 10685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:59.924014 10685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "5a0dec9e4aba4961b1f7b614c2e7439c"
format_stamp: "Formatted at 2026-09-27 10:49:59 on dist-test-slave-c60x"
server_key: "84152aa89b85877226adad5d21ba88bd"
server_key_iv: "aa9a6af0f162cc5c2092796fef5c09e0"
server_key_version: "encryptionkey@0"
I20260927 10:49:59.924126 10685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:59.935591 10685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:59.938411 10685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:59.938618 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:59.939281 10685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:49:59.940096 10685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:49:59.940181 10685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.940275 10685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:49:59.940327 10685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:59.949189 10685 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:38883
I20260927 10:49:59.949211 10812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:38883 every 8 connection(s)
I20260927 10:49:59.949824 10685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:49:59.951068 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 10685
I20260927 10:49:59.951225 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:49:59.951536 6270 external_mini_cluster.cc:1428] Setting key ae3f0082b1afad580c8787770b90a297
Sep 27 10:49:59 dist-test-slave-c60x krb5kdc[10313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:49:59.967900 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:59.951611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:57345 (local address 127.6.31.190:38351)
0927 10:49:59.951996 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:59.952001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:59.955607 (+ 3606us) server_negotiation.cc:407] Connection header received
0927 10:49:59.957186 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:59.957190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:59.957245 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:59.957331 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:59.958981 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.959608 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.960852 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.961054 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.961135 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:59.963943 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:59.963947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:59.963949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:59.963976 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:59.965843 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.966347 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.966350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.966351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.966425 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.966775 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.966779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.966780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.966978 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:59.967141 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:59.967660 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 10:49:59.967770 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260927 10:49:59.968716 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:59.952083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38351 (local address 127.6.31.131:57345)
0927 10:49:59.955458 (+ 3375us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:59.955519 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:59.956970 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:59.957595 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:59.957625 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:59.958176 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:59.958859 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:59.958876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.959783 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.959786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:59.960747 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:59.960757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.961188 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.961190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:59.961327 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:59.962232 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:59.962249 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:59.963768 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:59.965950 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:59.965955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:59.965967 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:59.966260 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:59.966523 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:59.966525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:59.966526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:59.966656 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:59.967180 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:59.967186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:59.967424 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 10:49:59.967606 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 10:49:59.967874 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3125,"thread_start_us":148,"threads_started":1}
I20260927 10:49:59.969812 10813 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38351
I20260927 10:49:59.970024 10813 heartbeater.cc:464] Registering TS with master...
I20260927 10:49:59.970415 10813 heartbeater.cc:511] Master 127.6.31.190:38351 requested a full tablet report, sending...
I20260927 10:49:59.971396 10356 ts_manager.cc:194] Registered new tserver with Master: 5a0dec9e4aba4961b1f7b614c2e7439c (127.6.31.131:38883)
I20260927 10:49:59.971957 10356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:57345
I20260927 10:49:59.977200 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:49:59.987722 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:59.979176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.6.31.190:38351)
0927 10:49:59.979338 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:59.979341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:59.979562 (+ 221us) server_negotiation.cc:407] Connection header received
0927 10:49:59.979717 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:59.979720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:59.979762 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:59.979855 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:59.980598 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.981177 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.982031 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:59.982366 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:59.982475 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:59.984630 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:59.984634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:59.984636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:59.984660 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:59.986142 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.986552 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.986555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.986557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.986634 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:59.986865 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:59.986868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:59.986869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:59.987073 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:59.987250 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:59.987437 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 10:49:59.987528 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 10:49:59.990442 10356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:49:59.993494 10356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:00.008457 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.002797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42523 (local address 127.0.0.1:49328)
0927 10:50:00.003488 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.003500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:00.003631 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:00.004278 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:00.004283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:00.004304 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:00.004402 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.004409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.005580 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.005587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.006787 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.006796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.007825 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.007829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.007938 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.007981 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 10:50:00.008145 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 10:50:00.008281 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":3,"spinlock_wait_cycles":7168,"thread_start_us":79,"threads_started":1}
I20260927 10:50:00.008505 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.002947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.6.31.129:42523)
0927 10:50:00.003898 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:00.003903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:00.003922 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:50:00.003982 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:00.003986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:00.004155 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:00.004277 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:00.004555 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.005457 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.006911 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.007700 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.007804 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.007997 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:00.008226 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 10:50:00.008333 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":73,"threads_started":1}
I20260927 10:50:00.009267 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.002196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38883 (local address 127.0.0.1:36692)
0927 10:50:00.003451 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.003500 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:00.003633 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:00.004091 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:00.004096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:00.004117 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:00.004249 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.004259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.005885 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.005890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.007277 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.007289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.008488 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.008497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.008941 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.008962 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:50:00.009035 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:50:00.009122 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":439,"spinlock_wait_cycles":10880,"thread_start_us":87,"threads_started":1}
I20260927 10:50:00.009608 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.002971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.6.31.130:35789)
0927 10:50:00.003370 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:00.003381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:00.003772 (+ 391us) server_negotiation.cc:407] Connection header received
0927 10:50:00.003850 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:00.003854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:00.004039 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:00.004187 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:00.004687 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.005800 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.007422 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.008159 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.008245 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.008360 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:00.009198 (+ 838us) server_negotiation.cc:299] Negotiation successful
0927 10:50:00.009308 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":119,"threads_started":1}
I20260927 10:50:00.009699 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.002309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.6.31.131:38883)
0927 10:50:00.002677 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:00.002688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:00.003669 (+ 981us) server_negotiation.cc:407] Connection header received
0927 10:50:00.003758 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:00.003766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:00.003931 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:00.004082 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:00.004375 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.005723 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.007405 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.008198 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.008300 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.008426 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:00.009453 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0927 10:50:00.009549 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":149,"threads_started":1}
I20260927 10:50:00.010042 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.002267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:35789 (local address 127.0.0.1:54602)
0927 10:50:00.003450 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.003500 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:00.003631 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:00.004348 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:00.004351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:00.004367 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:00.004510 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.004518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.005932 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.005935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.007292 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.007298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.008426 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.008430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.008538 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.008923 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 10:50:00.009038 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 10:50:00.009122 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":78,"spinlock_wait_cycles":296448,"thread_start_us":95,"threads_started":1}
I20260927 10:50:00.012234 10747 tablet_service.cc:1467] Processing CreateTablet for tablet 512e6e05b28142cfbad61a1e6bc07229 (DEFAULT_TABLE table=test-table [id=e0c50441728049958c8c7e5b71af7e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:00.012130 10481 tablet_service.cc:1467] Processing CreateTablet for tablet 512e6e05b28142cfbad61a1e6bc07229 (DEFAULT_TABLE table=test-table [id=e0c50441728049958c8c7e5b71af7e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:00.012557 10614 tablet_service.cc:1467] Processing CreateTablet for tablet 512e6e05b28142cfbad61a1e6bc07229 (DEFAULT_TABLE table=test-table [id=e0c50441728049958c8c7e5b71af7e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:00.013257 10481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512e6e05b28142cfbad61a1e6bc07229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:00.013283 10747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512e6e05b28142cfbad61a1e6bc07229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:00.013453 10614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512e6e05b28142cfbad61a1e6bc07229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:00.023243 10831 tablet_bootstrap.cc:492] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: Bootstrap starting.
I20260927 10:50:00.025122 10832 tablet_bootstrap.cc:492] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262: Bootstrap starting.
I20260927 10:50:00.026057 10831 tablet_bootstrap.cc:654] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:00.027277 10831 log.cc:824] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:00.027503 10832 tablet_bootstrap.cc:654] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:00.028155 10830 tablet_bootstrap.cc:492] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9: Bootstrap starting.
I20260927 10:50:00.028898 10832 log.cc:824] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:00.030604 10830 tablet_bootstrap.cc:654] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:00.031328 10832 tablet_bootstrap.cc:492] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262: No bootstrap required, opened a new log
I20260927 10:50:00.031535 10832 ts_tablet_manager.cc:1397] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:50:00.031842 10830 log.cc:824] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:00.033543 10831 tablet_bootstrap.cc:492] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: No bootstrap required, opened a new log
I20260927 10:50:00.033741 10831 ts_tablet_manager.cc:1397] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20260927 10:50:00.036092 10832 raft_consensus.cc:348] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } }
I20260927 10:50:00.036386 10832 raft_consensus.cc:374] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:00.036450 10832 raft_consensus.cc:729] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8242a44cb674199b402f82755d24262, State: Initialized, Role: FOLLOWER
I20260927 10:50:00.036921 10832 consensus_queue.cc:260] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } }
I20260927 10:50:00.037635 10547 heartbeater.cc:503] Master 127.6.31.190:38351 was elected leader, sending a full tablet report...
I20260927 10:50:00.037549 10831 raft_consensus.cc:348] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } }
I20260927 10:50:00.037807 10831 raft_consensus.cc:374] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:00.037864 10831 raft_consensus.cc:729] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0dec9e4aba4961b1f7b614c2e7439c, State: Initialized, Role: FOLLOWER
I20260927 10:50:00.037988 10832 ts_tablet_manager.cc:1428] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:50:00.039357 10830 tablet_bootstrap.cc:492] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9: No bootstrap required, opened a new log
I20260927 10:50:00.039356 10831 consensus_queue.cc:260] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } }
I20260927 10:50:00.039553 10830 ts_tablet_manager.cc:1397] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260927 10:50:00.040002 10813 heartbeater.cc:503] Master 127.6.31.190:38351 was elected leader, sending a full tablet report...
I20260927 10:50:00.040393 10831 ts_tablet_manager.cc:1428] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260927 10:50:00.043435 10830 raft_consensus.cc:348] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } }
I20260927 10:50:00.043722 10830 raft_consensus.cc:374] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:00.043762 10830 raft_consensus.cc:729] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c305edd9a48d43329a47daae3c7547d9, State: Initialized, Role: FOLLOWER
I20260927 10:50:00.044075 10830 consensus_queue.cc:260] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } }
I20260927 10:50:00.044612 10680 heartbeater.cc:503] Master 127.6.31.190:38351 was elected leader, sending a full tablet report...
I20260927 10:50:00.044919 10830 ts_tablet_manager.cc:1428] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 10:50:00.200749 10814 tablet.cc:2369] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:50:00.201748 10838 raft_consensus.cc:482] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:00.201963 10838 raft_consensus.cc:504] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } }
I20260927 10:50:00.203248 10838 leader_election.cc:288] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a0dec9e4aba4961b1f7b614c2e7439c (127.6.31.131:38883), b8242a44cb674199b402f82755d24262 (127.6.31.129:42523)
I20260927 10:50:00.207717 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.203421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38883 (local address 127.6.31.130:50621)
0927 10:50:00.203623 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.203641 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:00.203738 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:00.204149 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:00.204152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:00.204172 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:00.204314 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.204322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.205398 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.205404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.206302 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.206311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.207137 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.207141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.207398 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.207421 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:50:00.207517 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:50:00.207605 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":75}
I20260927 10:50:00.207765 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.203546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:50621 (local address 127.6.31.131:38883)
0927 10:50:00.203697 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:00.203704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:00.203725 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:50:00.203879 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:00.203883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:00.203952 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:00.204132 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:00.204447 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.205280 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.206436 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.206928 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.207074 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.207153 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:00.207558 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 10:50:00.207633 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:50:00.207765 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.203674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:43209 (local address 127.6.31.129:42523)
0927 10:50:00.203834 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:00.203839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:00.203989 (+ 150us) server_negotiation.cc:407] Connection header received
0927 10:50:00.204147 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:00.204150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:00.204201 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:00.204290 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:00.204615 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.205286 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.206545 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.206950 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.207128 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.207182 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:00.207558 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 10:50:00.207633 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:50:00.208132 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.203551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42523 (local address 127.6.31.130:43209)
0927 10:50:00.203889 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.203902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:00.204013 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:00.204311 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:00.204313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:00.204329 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:00.204467 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.204474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.205403 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.205409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.206393 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:00.206408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.207133 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.207137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:00.207318 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.207383 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 10:50:00.207897 (+ 514us) client_negotiation.cc:240] Negotiation successful
0927 10:50:00.207988 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":32,"thread_start_us":134,"threads_started":1}
I20260927 10:50:00.208803 10501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e6e05b28142cfbad61a1e6bc07229" candidate_uuid: "c305edd9a48d43329a47daae3c7547d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8242a44cb674199b402f82755d24262" is_pre_election: true
I20260927 10:50:00.209247 10501 raft_consensus.cc:2454] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c305edd9a48d43329a47daae3c7547d9 in term 0.
I20260927 10:50:00.209331 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e6e05b28142cfbad61a1e6bc07229" candidate_uuid: "c305edd9a48d43329a47daae3c7547d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" is_pre_election: true
I20260927 10:50:00.209647 10767 raft_consensus.cc:2454] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c305edd9a48d43329a47daae3c7547d9 in term 0.
I20260927 10:50:00.209750 10567 leader_election.cc:302] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8242a44cb674199b402f82755d24262, c305edd9a48d43329a47daae3c7547d9; no voters:
I20260927 10:50:00.210083 10838 raft_consensus.cc:2781] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:00.210177 10838 raft_consensus.cc:482] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:00.210213 10838 raft_consensus.cc:3037] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 0 FOLLOWER]: Advancing to term 1
W20260927 10:50:00.244397 10681 tablet.cc:2369] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:50:00.273252 10548 tablet.cc:2369] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:50:00.274355 10838 raft_consensus.cc:504] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } }
I20260927 10:50:00.274858 10838 leader_election.cc:288] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [CANDIDATE]: Term 1 election: Requested vote from peers 5a0dec9e4aba4961b1f7b614c2e7439c (127.6.31.131:38883), b8242a44cb674199b402f82755d24262 (127.6.31.129:42523)
I20260927 10:50:00.275431 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e6e05b28142cfbad61a1e6bc07229" candidate_uuid: "c305edd9a48d43329a47daae3c7547d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c"
I20260927 10:50:00.275434 10501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e6e05b28142cfbad61a1e6bc07229" candidate_uuid: "c305edd9a48d43329a47daae3c7547d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8242a44cb674199b402f82755d24262"
I20260927 10:50:00.275584 10501 raft_consensus.cc:3037] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:00.275588 10767 raft_consensus.cc:3037] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:00.276726 10767 raft_consensus.cc:2454] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c305edd9a48d43329a47daae3c7547d9 in term 1.
I20260927 10:50:00.277200 10570 leader_election.cc:302] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0dec9e4aba4961b1f7b614c2e7439c, c305edd9a48d43329a47daae3c7547d9; no voters:
I20260927 10:50:00.277274 10501 raft_consensus.cc:2454] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c305edd9a48d43329a47daae3c7547d9 in term 1.
I20260927 10:50:00.277433 10838 raft_consensus.cc:2781] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:00.277658 10838 raft_consensus.cc:686] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 1 LEADER]: Becoming Leader. State: Replica: c305edd9a48d43329a47daae3c7547d9, State: Running, Role: LEADER
I20260927 10:50:00.278023 10838 consensus_queue.cc:237] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } }
I20260927 10:50:00.281266 10356 catalog_manager.cc:5495] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c305edd9a48d43329a47daae3c7547d9 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "c305edd9a48d43329a47daae3c7547d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c305edd9a48d43329a47daae3c7547d9" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 } health_report { overall_health: UNKNOWN } } }
I20260927 10:50:00.311985 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.308303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.6.31.130:35789)
0927 10:50:00.308451 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:00.308455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:00.308468 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:50:00.308564 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:00.308566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:00.308621 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:00.308718 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:00.309065 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.309612 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.310436 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:00.310636 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:00.310719 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:00.310988 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:00.311483 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:00.311597 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:00.311723 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 10:50:00.311774 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:50:00.318691 10501 raft_consensus.cc:1261] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [term 1 FOLLOWER]: Refusing update from remote peer c305edd9a48d43329a47daae3c7547d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:00.318755 10767 raft_consensus.cc:1261] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [term 1 FOLLOWER]: Refusing update from remote peer c305edd9a48d43329a47daae3c7547d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:00.319396 10840 consensus_queue.cc:1035] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8242a44cb674199b402f82755d24262" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:00.319597 10838 consensus_queue.cc:1035] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0dec9e4aba4961b1f7b614c2e7439c" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:00.327152 10845 mvcc.cc:205] Tried to move back new op lower bound from 7333913396497223680 to 7333913396341993472. Current Snapshot: MvccSnapshot[applied={T|T < 7333913396497223680}]
I20260927 10:50:00.327704 10844 mvcc.cc:205] Tried to move back new op lower bound from 7333913396497223680 to 7333913396341993472. Current Snapshot: MvccSnapshot[applied={T|T < 7333913396497223680}]
I20260927 10:50:00.335038 10357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:00.335251 10357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:00.337811 10357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d7c1fbd9c0e4db4b9398281f81c02a9: Sending DeleteTablet for 3 replicas of tablet 512e6e05b28142cfbad61a1e6bc07229
I20260927 10:50:00.338610 10747 tablet_service.cc:1514] Processing DeleteTablet for tablet 512e6e05b28142cfbad61a1e6bc07229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:00 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:36692
I20260927 10:50:00.338646 10614 tablet_service.cc:1514] Processing DeleteTablet for tablet 512e6e05b28142cfbad61a1e6bc07229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:00 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:54602
I20260927 10:50:00.339152 10853 tablet_replica.cc:330] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9: stopping tablet replica
I20260927 10:50:00.339164 10481 tablet_service.cc:1514] Processing DeleteTablet for tablet 512e6e05b28142cfbad61a1e6bc07229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:00 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:49328
I20260927 10:50:00.339465 10853 raft_consensus.cc:2229] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:50:00.339707 10855 tablet_replica.cc:330] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262: stopping tablet replica
I20260927 10:50:00.339834 10853 raft_consensus.cc:2258] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:00.340086 10855 raft_consensus.cc:2229] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:00.340194 10854 tablet_replica.cc:330] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: stopping tablet replica
I20260927 10:50:00.340468 10854 raft_consensus.cc:2229] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:00.340476 10855 raft_consensus.cc:2258] T 512e6e05b28142cfbad61a1e6bc07229 P b8242a44cb674199b402f82755d24262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:00.340561 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 10419
I20260927 10:50:00.340751 10854 raft_consensus.cc:2258] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:00.341688 10853 ts_tablet_manager.cc:1905] T 512e6e05b28142cfbad61a1e6bc07229 P c305edd9a48d43329a47daae3c7547d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:00.342406 10854 ts_tablet_manager.cc:1905] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:50:00.347847 10341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:42523 (error 108)
I20260927 10:50:00.348726 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 10552
I20260927 10:50:00.348977 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.348634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42523 (local address 127.0.0.1:42202)
0927 10:50:00.348785 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.348866 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 10:50:00.349475 10341 catalog_manager.cc:4584] TS b8242a44cb674199b402f82755d24262 (127.6.31.129:42523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 512e6e05b28142cfbad61a1e6bc07229: Network error: Client connection negotiation failed: client connection to 127.6.31.129:42523: connect: Connection refused (error 111)
I20260927 10:50:00.379801 10854 ts_tablet_manager.cc:1918] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:00.379951 10854 log.cc:1196] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/512e6e05b28142cfbad61a1e6bc07229
I20260927 10:50:00.380334 10854 ts_tablet_manager.cc:1939] T 512e6e05b28142cfbad61a1e6bc07229 P 5a0dec9e4aba4961b1f7b614c2e7439c: Deleting consensus metadata
I20260927 10:50:00.382831 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 10685
I20260927 10:50:00.384598 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.383072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:35789 (local address 127.0.0.1:53448)
0927 10:50:00.384461 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.384519 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.130:35789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1289}
I20260927 10:50:00.402604 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.402302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42523 (local address 127.0.0.1:42206)
0927 10:50:00.402463 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.402521 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 10:50:00.403393 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.403173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42523 (local address 127.0.0.1:42216)
0927 10:50:00.403278 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.403337 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 10:50:00.429035 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.428680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:35789 (local address 127.0.0.1:53458)
0927 10:50:00.428846 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.428914 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.130:35789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:50:00.429791 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.429573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:35789 (local address 127.0.0.1:53460)
0927 10:50:00.429688 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.429740 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.130:35789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:50:00.440668 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 10326
2026-09-27T10:50:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2555 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T10:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:00Z Disabled control of system clock
I20260927 10:50:00.470285 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46557
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:46557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:50:00.580363 10860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:00.580646 10860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:00.580721 10860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:00.584684 10860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:00.584764 10860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:00.584791 10860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:00.584846 10860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:00.584903 10860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:00.589622 10860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:46557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:00.590643 10860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:00.592000 10860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:00.600309 10868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:00.600342 10865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:00.600342 10866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:00.600485 10860 server_base.cc:1034] running on GCE node
I20260927 10:50:00.601106 10860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:00.602182 10860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:00.603338 10860 hybrid_clock.cc:648] HybridClock initialized: now 1790506200603309 us; error 40 us; skew 500 ppm
I20260927 10:50:00.605432 10860 webserver.cc:460] Webserver started at http://127.6.31.190:42413/ using document root <none> and password file <none>
I20260927 10:50:00.606083 10860 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:00.606156 10860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:00.606374 10860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:00.608477 10860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "1e9e45c9d6cc4b1888c6144f20aa959a"
format_stamp: "Formatted at 2026-09-27 10:50:00 on dist-test-slave-c60x"
server_key: "1347cadf0bb5486e077238e4b75d9e31"
server_key_iv: "53972f72082e106fd9174ebb8d49215c"
server_key_version: "encryptionkey@0"
I20260927 10:50:00.609023 10860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "1e9e45c9d6cc4b1888c6144f20aa959a"
format_stamp: "Formatted at 2026-09-27 10:50:00 on dist-test-slave-c60x"
server_key: "1347cadf0bb5486e077238e4b75d9e31"
server_key_iv: "53972f72082e106fd9174ebb8d49215c"
server_key_version: "encryptionkey@0"
I20260927 10:50:00.612555 10860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:00.614781 10874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:00.616019 10860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:00.616182 10860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "1e9e45c9d6cc4b1888c6144f20aa959a"
format_stamp: "Formatted at 2026-09-27 10:50:00 on dist-test-slave-c60x"
server_key: "1347cadf0bb5486e077238e4b75d9e31"
server_key_iv: "53972f72082e106fd9174ebb8d49215c"
server_key_version: "encryptionkey@0"
I20260927 10:50:00.616297 10860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:00.620772 10860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:00.621372 10860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:00.621555 10860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:00.628595 10860 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:46557
I20260927 10:50:00.628623 10926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:46557 every 8 connection(s)
I20260927 10:50:00.629504 10860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:00.632719 10927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:00.634214 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 10860
I20260927 10:50:00.634341 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:00.634615 6270 external_mini_cluster.cc:1428] Setting key 396de0f5219f62442d5812ce9d77b41b
I20260927 10:50:00.639876 10927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a: Bootstrap starting.
I20260927 10:50:00.640316 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.636059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.6.31.190:46557)
0927 10:50:00.636691 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:00.636706 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:00.636741 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:50:00.637389 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:00.637405 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:00.637744 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:00.638060 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:00.638888 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:00.638893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:00.638904 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:00.638914 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:00.638969 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:00.638985 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:00.639156 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:00.639322 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:50:00.639566 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
I20260927 10:50:00.642369 10927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:00.643188 10927 log.cc:824] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:00.645072 10927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a: No bootstrap required, opened a new log
I20260927 10:50:00.648773 10927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9e45c9d6cc4b1888c6144f20aa959a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46557 } }
I20260927 10:50:00.649071 10927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:00.649137 10927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9e45c9d6cc4b1888c6144f20aa959a, State: Initialized, Role: FOLLOWER
I20260927 10:50:00.649606 10927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9e45c9d6cc4b1888c6144f20aa959a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46557 } }
I20260927 10:50:00.649770 10927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:00.649866 10927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:00.649973 10927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:00.678645 10927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9e45c9d6cc4b1888c6144f20aa959a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46557 } }
I20260927 10:50:00.679234 10927 leader_election.cc:302] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e9e45c9d6cc4b1888c6144f20aa959a; no voters:
I20260927 10:50:00.679659 10927 leader_election.cc:288] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:00.679786 10932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:00.680037 10932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [term 1 LEADER]: Becoming Leader. State: Replica: 1e9e45c9d6cc4b1888c6144f20aa959a, State: Running, Role: LEADER
I20260927 10:50:00.680385 10932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9e45c9d6cc4b1888c6144f20aa959a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46557 } }
I20260927 10:50:00.681169 10927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:00.681906 10934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e9e45c9d6cc4b1888c6144f20aa959a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9e45c9d6cc4b1888c6144f20aa959a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46557 } } }
I20260927 10:50:00.682026 10934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:00.682418 10933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e9e45c9d6cc4b1888c6144f20aa959a. Latest consensus state: current_term: 1 leader_uuid: "1e9e45c9d6cc4b1888c6144f20aa959a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9e45c9d6cc4b1888c6144f20aa959a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46557 } } }
I20260927 10:50:00.682564 10933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:00.683022 10937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:00.685999 10937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:00.691362 10937 catalog_manager.cc:1347] Generated new cluster ID: b248f9bbb0da4f7abb8120fd16eea83c
I20260927 10:50:00.691437 10937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:00.702003 10937 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:00.702319 10937 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:00.708725 10937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a: Generated new TSK 0
I20260927 10:50:00.709406 10937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:50:00.723573 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46557
--builtin_ntp_servers=127.6.31.148:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:50:00.841490 10951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:00.841763 10951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:00.841807 10951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:00.846030 10951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:00.846101 10951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:00.846163 10951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:00.851262 10951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:00.852588 10951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:00.854144 10951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:00.864471 10956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:00.864463 10957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:00.864463 10959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:00.864712 10951 server_base.cc:1034] running on GCE node
I20260927 10:50:00.865263 10951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:00.865942 10951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:00.867206 10951 hybrid_clock.cc:648] HybridClock initialized: now 1790506200867185 us; error 34 us; skew 500 ppm
I20260927 10:50:00.869573 10951 webserver.cc:460] Webserver started at http://127.6.31.129:41859/ using document root <none> and password file <none>
I20260927 10:50:00.870205 10951 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:00.870301 10951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:00.870518 10951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:00.872780 10951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "1bc2da5b644a486da43d7a5d3cd80824"
format_stamp: "Formatted at 2026-09-27 10:50:00 on dist-test-slave-c60x"
server_key: "13ae68fc4b1c1021a203a3d23ead5835"
server_key_iv: "8000455e20fd4afb174a4dfd1186c2bc"
server_key_version: "encryptionkey@0"
I20260927 10:50:00.873373 10951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "1bc2da5b644a486da43d7a5d3cd80824"
format_stamp: "Formatted at 2026-09-27 10:50:00 on dist-test-slave-c60x"
server_key: "13ae68fc4b1c1021a203a3d23ead5835"
server_key_iv: "8000455e20fd4afb174a4dfd1186c2bc"
server_key_version: "encryptionkey@0"
I20260927 10:50:00.877187 10951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 10:50:00.879576 10965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:00.880872 10951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:00.881012 10951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "1bc2da5b644a486da43d7a5d3cd80824"
format_stamp: "Formatted at 2026-09-27 10:50:00 on dist-test-slave-c60x"
server_key: "13ae68fc4b1c1021a203a3d23ead5835"
server_key_iv: "8000455e20fd4afb174a4dfd1186c2bc"
server_key_version: "encryptionkey@0"
I20260927 10:50:00.881152 10951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:00.886221 10951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:00.886864 10951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:00.887060 10951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:00.887707 10951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:00.888640 10951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:00.888708 10951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:00.888772 10951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:00.888816 10951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:00.897882 10951 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:32875
I20260927 10:50:00.897903 11078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:32875 every 8 connection(s)
I20260927 10:50:00.898838 10951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:00.899991 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 10951
I20260927 10:50:00.900118 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:50:00.900442 6270 external_mini_cluster.cc:1428] Setting key 398442d661363a0b882989f81487721f
I20260927 10:50:00.902400 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46557
--builtin_ntp_servers=127.6.31.148:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 10:50:00.905694 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.900913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:36487 (local address 127.6.31.190:46557)
0927 10:50:00.901054 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:00.901059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:00.902274 (+ 1215us) server_negotiation.cc:407] Connection header received
0927 10:50:00.903743 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:00.903746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:00.903789 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:00.903866 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:00.905183 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:00.905187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:00.905189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:00.905207 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:00.905227 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:00.905235 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:00.905332 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:00.905507 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 10:50:00.905555 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:50:00.906769 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:00.901380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46557 (local address 127.6.31.129:36487)
0927 10:50:00.902105 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:00.902157 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:00.903529 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:00.904031 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:00.904042 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:00.904152 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:00.904868 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:00.904870 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:00.904920 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:00.904992 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:00.905325 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:00.905333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:00.905340 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:50:00.905548 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 10:50:00.905798 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":137,"threads_started":1}
I20260927 10:50:00.908253 11079 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46557
I20260927 10:50:00.908509 11079 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:00.908950 11079 heartbeater.cc:511] Master 127.6.31.190:46557 requested a full tablet report, sending...
I20260927 10:50:00.910573 10891 ts_manager.cc:194] Registered new tserver with Master: 1bc2da5b644a486da43d7a5d3cd80824 (127.6.31.129:32875)
W20260927 10:50:01.013706 11082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:01.013999 11082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:01.014074 11082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:01.018204 11082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:01.018280 11082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:01.018368 11082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:01.023450 11082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:01.024569 11082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:01.025916 11082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:01.035359 11090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:01.035423 11088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:01.035404 11087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:01.035521 11082 server_base.cc:1034] running on GCE node
I20260927 10:50:01.035897 11082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:01.036515 11082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:01.037667 11082 hybrid_clock.cc:648] HybridClock initialized: now 1790506201037634 us; error 44 us; skew 500 ppm
I20260927 10:50:01.039525 11082 webserver.cc:460] Webserver started at http://127.6.31.130:40933/ using document root <none> and password file <none>
I20260927 10:50:01.040079 11082 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:01.040148 11082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:01.040360 11082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:01.042377 11082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "37b45b19704e4cdd8197ee0e25f6257a"
format_stamp: "Formatted at 2026-09-27 10:50:01 on dist-test-slave-c60x"
server_key: "19c4b75be1f544d52ca18eab79a64820"
server_key_iv: "b6632f27da51174d03367ac4a9ed6678"
server_key_version: "encryptionkey@0"
I20260927 10:50:01.042920 11082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "37b45b19704e4cdd8197ee0e25f6257a"
format_stamp: "Formatted at 2026-09-27 10:50:01 on dist-test-slave-c60x"
server_key: "19c4b75be1f544d52ca18eab79a64820"
server_key_iv: "b6632f27da51174d03367ac4a9ed6678"
server_key_version: "encryptionkey@0"
I20260927 10:50:01.046550 11082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:50:01.048882 11096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:01.050096 11082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:50:01.050225 11082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "37b45b19704e4cdd8197ee0e25f6257a"
format_stamp: "Formatted at 2026-09-27 10:50:01 on dist-test-slave-c60x"
server_key: "19c4b75be1f544d52ca18eab79a64820"
server_key_iv: "b6632f27da51174d03367ac4a9ed6678"
server_key_version: "encryptionkey@0"
I20260927 10:50:01.050329 11082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:01.055049 11082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:01.055687 11082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:01.055888 11082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:01.056475 11082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:01.057396 11082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:01.057451 11082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:01.057489 11082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:01.057543 11082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:01.067268 11082 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:35343
I20260927 10:50:01.067291 11209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:35343 every 8 connection(s)
I20260927 10:50:01.068151 11082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:01.068518 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 11082
I20260927 10:50:01.068655 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:50:01.068964 6270 external_mini_cluster.cc:1428] Setting key 33ee9d71cbdf6eff068ba481538c620a
I20260927 10:50:01.071242 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46557
--builtin_ntp_servers=127.6.31.148:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 10:50:01.075035 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.069893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:38355 (local address 127.6.31.190:46557)
0927 10:50:01.070036 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.070041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.071388 (+ 1347us) server_negotiation.cc:407] Connection header received
0927 10:50:01.073279 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.073285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.073354 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.073453 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.074511 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.074518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.074524 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.074541 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.074567 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.074580 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.074681 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.074801 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.074851 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:50:01.075889 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.070442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46557 (local address 127.6.31.130:38355)
0927 10:50:01.071230 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.071284 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:01.073011 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:01.073598 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:01.073604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:01.073689 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:01.074268 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:01.074269 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:01.074307 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:01.074372 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:01.074668 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:01.074674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:01.074678 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:50:01.074803 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 10:50:01.075022 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260927 10:50:01.077126 11210 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46557
I20260927 10:50:01.077404 11210 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:01.077945 11210 heartbeater.cc:511] Master 127.6.31.190:46557 requested a full tablet report, sending...
I20260927 10:50:01.079236 10891 ts_manager.cc:194] Registered new tserver with Master: 37b45b19704e4cdd8197ee0e25f6257a (127.6.31.130:35343)
W20260927 10:50:01.196077 11213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:01.196398 11213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:01.196481 11213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:01.201237 11213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:01.201328 11213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:01.201421 11213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:01.206650 11213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:01.207949 11213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:01.209447 11213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:50:01.219542 11213 server_base.cc:1034] running on GCE node
W20260927 10:50:01.219447 11218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:01.219471 11219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:01.219502 11221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:01.220320 11213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:01.221125 11213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:01.222337 11213 hybrid_clock.cc:648] HybridClock initialized: now 1790506201222296 us; error 48 us; skew 500 ppm
I20260927 10:50:01.224419 11213 webserver.cc:460] Webserver started at http://127.6.31.131:37965/ using document root <none> and password file <none>
I20260927 10:50:01.225037 11213 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:01.225100 11213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:01.225314 11213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:01.227732 11213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "7260323f0d594b71a8f8d77ea70b6e4b"
format_stamp: "Formatted at 2026-09-27 10:50:01 on dist-test-slave-c60x"
server_key: "b14d930c38220af77eb7f8f73e617486"
server_key_iv: "70bed54877eee45bafcb127089a016e0"
server_key_version: "encryptionkey@0"
I20260927 10:50:01.228322 11213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "7260323f0d594b71a8f8d77ea70b6e4b"
format_stamp: "Formatted at 2026-09-27 10:50:01 on dist-test-slave-c60x"
server_key: "b14d930c38220af77eb7f8f73e617486"
server_key_iv: "70bed54877eee45bafcb127089a016e0"
server_key_version: "encryptionkey@0"
I20260927 10:50:01.232319 11213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:50:01.234817 11227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:01.236159 11213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:01.236258 11213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "7260323f0d594b71a8f8d77ea70b6e4b"
format_stamp: "Formatted at 2026-09-27 10:50:01 on dist-test-slave-c60x"
server_key: "b14d930c38220af77eb7f8f73e617486"
server_key_iv: "70bed54877eee45bafcb127089a016e0"
server_key_version: "encryptionkey@0"
I20260927 10:50:01.236346 11213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:01.241003 11213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:01.241572 11213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:01.241709 11213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:01.242204 11213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:01.243064 11213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:01.243181 11213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:01.243223 11213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:01.243245 11213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:01.252446 11213 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:37631
I20260927 10:50:01.252487 11340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:37631 every 8 connection(s)
I20260927 10:50:01.253439 11213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:50:01.258198 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 11213
I20260927 10:50:01.258322 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:50:01.258610 6270 external_mini_cluster.cc:1428] Setting key 9b67b926120820dd549dd2dd144b5eac
I20260927 10:50:01.260901 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.255502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:41115 (local address 127.6.31.190:46557)
0927 10:50:01.255658 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.255662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.256700 (+ 1038us) server_negotiation.cc:407] Connection header received
0927 10:50:01.257925 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.257929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.257965 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.258099 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.260423 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.260426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.260428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.260436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.260453 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.260461 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.260569 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.260725 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.260769 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:50:01.261765 11343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.255893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46557 (local address 127.6.31.131:41115)
0927 10:50:01.256528 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.256571 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:01.257744 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:01.258261 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:01.258267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:01.258402 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:01.259261 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:01.259263 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:01.259315 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:01.259405 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:01.260560 (+ 1155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:01.260568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:01.260575 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:50:01.260759 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 10:50:01.261029 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260927 10:50:01.262827 11341 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46557
I20260927 10:50:01.263044 11341 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:01.263448 11341 heartbeater.cc:511] Master 127.6.31.190:46557 requested a full tablet report, sending...
I20260927 10:50:01.264387 10891 ts_manager.cc:194] Registered new tserver with Master: 7260323f0d594b71a8f8d77ea70b6e4b (127.6.31.131:37631)
I20260927 10:50:01.272784 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:50:01.276427 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.274699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.6.31.190:46557)
0927 10:50:01.274870 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.274875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.275073 (+ 198us) server_negotiation.cc:407] Connection header received
0927 10:50:01.275175 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.275178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.275218 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.275313 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.276009 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.276020 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.276023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.276041 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.276060 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.276067 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.276161 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.276238 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.276278 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 10:50:01.279170 10891 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:50:01.282413 10891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:01.296648 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.293705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:32875 (local address 127.0.0.1:38462)
0927 10:50:01.294525 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.294535 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:01.294585 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:01.294973 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:01.294979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:01.295051 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:01.295948 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:01.295951 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:01.295974 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:01.296003 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:01.296330 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:01.296335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:01.296338 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:50:01.296398 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 10:50:01.296464 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":94,"threads_started":1}
I20260927 10:50:01.297174 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.294099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.6.31.129:32875)
0927 10:50:01.294576 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.294585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.294606 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:50:01.294650 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.294654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.294825 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.294932 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.296130 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.296135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.296145 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.296156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.296192 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.296220 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.296876 (+ 656us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.296925 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.297003 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":135,"threads_started":1}
I20260927 10:50:01.296655 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.292997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:37631 (local address 127.0.0.1:48974)
0927 10:50:01.294042 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.294093 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:01.294380 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:01.295172 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:01.295177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:01.295197 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:01.295786 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:01.295789 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:01.295810 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:01.295879 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:01.296293 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:01.296303 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:01.296311 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:50:01.296441 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 10:50:01.296483 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"spinlock_wait_cycles":2432,"thread_start_us":109,"threads_started":1}
I20260927 10:50:01.297214 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.293456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.6.31.131:37631)
0927 10:50:01.294774 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.294785 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.294802 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:50:01.294842 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.294845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.295024 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.295181 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.296013 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.296019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.296031 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.296050 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.296114 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.296153 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.296315 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.296965 (+ 650us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.297050 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234,"thread_start_us":161,"threads_started":1}
I20260927 10:50:01.298748 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.293445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:35343 (local address 127.0.0.1:45644)
0927 10:50:01.294042 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.294093 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:01.294332 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:01.296357 (+ 2025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:01.296363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:01.296381 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:01.296963 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:01.296965 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:01.296975 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:01.297001 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:01.298488 (+ 1487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:01.298491 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:01.298494 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:50:01.298580 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 10:50:01.298623 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"spinlock_wait_cycles":296064,"thread_start_us":89,"threads_started":1}
I20260927 10:50:01.298812 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.293622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.6.31.130:35343)
0927 10:50:01.295944 (+ 2322us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.295956 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.295990 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:50:01.296040 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.296048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.296204 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.296600 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.298332 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.298336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.298343 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.298350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.298380 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.298393 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.298515 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.298593 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.298658 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2201,"thread_start_us":109,"threads_started":1}
I20260927 10:50:01.300287 11013 tablet_service.cc:1467] Processing CreateTablet for tablet df245fefde974944bec876c3150b886d (DEFAULT_TABLE table=test-table [id=af769d9b23ed4395aece1fbd9c3f55b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:01.300282 11275 tablet_service.cc:1467] Processing CreateTablet for tablet df245fefde974944bec876c3150b886d (DEFAULT_TABLE table=test-table [id=af769d9b23ed4395aece1fbd9c3f55b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:01.301422 11013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df245fefde974944bec876c3150b886d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:01.301422 11275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df245fefde974944bec876c3150b886d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:01.301187 11144 tablet_service.cc:1467] Processing CreateTablet for tablet df245fefde974944bec876c3150b886d (DEFAULT_TABLE table=test-table [id=af769d9b23ed4395aece1fbd9c3f55b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:01.301968 11144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df245fefde974944bec876c3150b886d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:01.308013 11358 tablet_bootstrap.cc:492] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a: Bootstrap starting.
I20260927 10:50:01.311305 11358 tablet_bootstrap.cc:654] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:01.311481 11359 tablet_bootstrap.cc:492] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824: Bootstrap starting.
I20260927 10:50:01.312114 11358 log.cc:824] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:01.312882 11360 tablet_bootstrap.cc:492] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: Bootstrap starting.
I20260927 10:50:01.313469 11359 tablet_bootstrap.cc:654] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:01.314278 11359 log.cc:824] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:01.314344 11358 tablet_bootstrap.cc:492] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a: No bootstrap required, opened a new log
I20260927 10:50:01.314512 11358 ts_tablet_manager.cc:1397] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260927 10:50:01.314762 11360 tablet_bootstrap.cc:654] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:01.315583 11360 log.cc:824] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:01.316807 11359 tablet_bootstrap.cc:492] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824: No bootstrap required, opened a new log
I20260927 10:50:01.316989 11359 ts_tablet_manager.cc:1397] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 10:50:01.317231 11360 tablet_bootstrap.cc:492] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: No bootstrap required, opened a new log
I20260927 10:50:01.317416 11360 ts_tablet_manager.cc:1397] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:50:01.319073 11358 raft_consensus.cc:348] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } }
I20260927 10:50:01.319388 11358 raft_consensus.cc:374] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:01.319442 11358 raft_consensus.cc:729] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b45b19704e4cdd8197ee0e25f6257a, State: Initialized, Role: FOLLOWER
I20260927 10:50:01.320010 11358 consensus_queue.cc:260] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } }
I20260927 10:50:01.320731 11210 heartbeater.cc:503] Master 127.6.31.190:46557 was elected leader, sending a full tablet report...
I20260927 10:50:01.321228 11360 raft_consensus.cc:348] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } }
I20260927 10:50:01.321473 11360 raft_consensus.cc:374] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:01.321537 11360 raft_consensus.cc:729] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7260323f0d594b71a8f8d77ea70b6e4b, State: Initialized, Role: FOLLOWER
I20260927 10:50:01.321517 11359 raft_consensus.cc:348] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } }
I20260927 10:50:01.321767 11359 raft_consensus.cc:374] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:01.321822 11359 raft_consensus.cc:729] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc2da5b644a486da43d7a5d3cd80824, State: Initialized, Role: FOLLOWER
I20260927 10:50:01.322048 11360 consensus_queue.cc:260] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } }
I20260927 10:50:01.322201 11359 consensus_queue.cc:260] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } }
I20260927 10:50:01.322793 11079 heartbeater.cc:503] Master 127.6.31.190:46557 was elected leader, sending a full tablet report...
I20260927 10:50:01.323055 11360 ts_tablet_manager.cc:1428] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:50:01.323158 11359 ts_tablet_manager.cc:1428] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 10:50:01.323377 11341 heartbeater.cc:503] Master 127.6.31.190:46557 was elected leader, sending a full tablet report...
I20260927 10:50:01.325120 11358 ts_tablet_manager.cc:1428] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260927 10:50:01.388442 11364 raft_consensus.cc:482] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:01.388708 11364 raft_consensus.cc:504] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } }
I20260927 10:50:01.389871 11364 leader_election.cc:288] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7260323f0d594b71a8f8d77ea70b6e4b (127.6.31.131:37631), 1bc2da5b644a486da43d7a5d3cd80824 (127.6.31.129:32875)
I20260927 10:50:01.391772 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.390108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:37631 (local address 127.6.31.130:51407)
0927 10:50:01.390257 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.390273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:01.390347 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:01.390615 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:01.390618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:01.390634 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:01.391187 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:01.391189 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:01.391201 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:01.391231 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:01.391504 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:01.391506 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:01.391509 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:50:01.391594 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:50:01.391634 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:50:01.391769 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.390232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:51407 (local address 127.6.31.131:37631)
0927 10:50:01.390355 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.390362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.390379 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:50:01.390428 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.390434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.390473 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.390595 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.391336 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.391340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.391342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.391354 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.391381 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.391390 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.391494 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.391584 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.391629 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:50:01.392238 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.390527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:52893 (local address 127.6.31.129:32875)
0927 10:50:01.390655 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.390659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.390942 (+ 283us) server_negotiation.cc:407] Connection header received
0927 10:50:01.391035 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.391039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.391082 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.391199 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.391862 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.391865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.391868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.391879 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.391899 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.391911 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.392013 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.392073 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.392111 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:50:01.392395 11295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df245fefde974944bec876c3150b886d" candidate_uuid: "37b45b19704e4cdd8197ee0e25f6257a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" is_pre_election: true
I20260927 10:50:01.392705 11295 raft_consensus.cc:2454] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37b45b19704e4cdd8197ee0e25f6257a in term 0.
I20260927 10:50:01.392993 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.390395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:32875 (local address 127.6.31.130:52893)
0927 10:50:01.390839 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.390856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:01.390946 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:01.391227 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:01.391230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:01.391249 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:01.391706 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:01.391708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:01.391717 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:01.391740 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:01.391993 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:01.391996 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:01.392001 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:50:01.392075 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:50:01.392104 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":129,"threads_started":1}
I20260927 10:50:01.393169 11098 leader_election.cc:302] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b45b19704e4cdd8197ee0e25f6257a, 7260323f0d594b71a8f8d77ea70b6e4b; no voters:
I20260927 10:50:01.393411 11364 raft_consensus.cc:2781] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:01.393518 11364 raft_consensus.cc:482] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:01.393568 11364 raft_consensus.cc:3037] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:01.393824 11033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df245fefde974944bec876c3150b886d" candidate_uuid: "37b45b19704e4cdd8197ee0e25f6257a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bc2da5b644a486da43d7a5d3cd80824" is_pre_election: true
I20260927 10:50:01.394073 11033 raft_consensus.cc:2454] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37b45b19704e4cdd8197ee0e25f6257a in term 0.
W20260927 10:50:01.400027 11080 tablet.cc:2369] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:50:01.484724 11364 raft_consensus.cc:504] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } }
I20260927 10:50:01.485291 11364 leader_election.cc:288] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [CANDIDATE]: Term 1 election: Requested vote from peers 7260323f0d594b71a8f8d77ea70b6e4b (127.6.31.131:37631), 1bc2da5b644a486da43d7a5d3cd80824 (127.6.31.129:32875)
I20260927 10:50:01.485788 11295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df245fefde974944bec876c3150b886d" candidate_uuid: "37b45b19704e4cdd8197ee0e25f6257a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7260323f0d594b71a8f8d77ea70b6e4b"
I20260927 10:50:01.485906 11033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df245fefde974944bec876c3150b886d" candidate_uuid: "37b45b19704e4cdd8197ee0e25f6257a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bc2da5b644a486da43d7a5d3cd80824"
I20260927 10:50:01.485986 11295 raft_consensus.cc:3037] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:01.486054 11033 raft_consensus.cc:3037] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:01.487251 11295 raft_consensus.cc:2454] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37b45b19704e4cdd8197ee0e25f6257a in term 1.
I20260927 10:50:01.487651 11098 leader_election.cc:302] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b45b19704e4cdd8197ee0e25f6257a, 7260323f0d594b71a8f8d77ea70b6e4b; no voters:
I20260927 10:50:01.487739 11033 raft_consensus.cc:2454] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37b45b19704e4cdd8197ee0e25f6257a in term 1.
I20260927 10:50:01.487890 11364 raft_consensus.cc:2781] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:01.488132 11364 raft_consensus.cc:686] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 1 LEADER]: Becoming Leader. State: Replica: 37b45b19704e4cdd8197ee0e25f6257a, State: Running, Role: LEADER
I20260927 10:50:01.488525 11364 consensus_queue.cc:237] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } }
I20260927 10:50:01.491765 10889 catalog_manager.cc:5495] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37b45b19704e4cdd8197ee0e25f6257a (127.6.31.130). New cstate: current_term: 1 leader_uuid: "37b45b19704e4cdd8197ee0e25f6257a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b45b19704e4cdd8197ee0e25f6257a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 35343 } health_report { overall_health: HEALTHY } } }
W20260927 10:50:01.504395 11342 tablet.cc:2369] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:50:01.532301 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.530794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.6.31.130:35343)
0927 10:50:01.530935 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.530942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.530958 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:50:01.531040 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.531042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.531156 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.531244 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.531897 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.531901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.531903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.531915 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.531936 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.531952 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.532043 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.532123 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.532163 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:50:01.538030 11295 raft_consensus.cc:1261] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [term 1 FOLLOWER]: Refusing update from remote peer 37b45b19704e4cdd8197ee0e25f6257a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:01.538031 11033 raft_consensus.cc:1261] T df245fefde974944bec876c3150b886d P 1bc2da5b644a486da43d7a5d3cd80824 [term 1 FOLLOWER]: Refusing update from remote peer 37b45b19704e4cdd8197ee0e25f6257a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:01.538772 11364 consensus_queue.cc:1035] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc2da5b644a486da43d7a5d3cd80824" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:01.539016 11368 consensus_queue.cc:1035] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7260323f0d594b71a8f8d77ea70b6e4b" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:01.546061 11371 mvcc.cc:205] Tried to move back new op lower bound from 7333913401492574208 to 7333913401299853312. Current Snapshot: MvccSnapshot[applied={T|T < 7333913401492574208}]
I20260927 10:50:01.546456 11372 mvcc.cc:205] Tried to move back new op lower bound from 7333913401492574208 to 7333913401299853312. Current Snapshot: MvccSnapshot[applied={T|T < 7333913401492574208}]
I20260927 10:50:01.549477 11373 mvcc.cc:205] Tried to move back new op lower bound from 7333913401492574208 to 7333913401299853312. Current Snapshot: MvccSnapshot[applied={T|T < 7333913401492574208}]
I20260927 10:50:01.555022 10890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:01.555236 10890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:01.557729 10890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e9e45c9d6cc4b1888c6144f20aa959a: Sending DeleteTablet for 3 replicas of tablet df245fefde974944bec876c3150b886d
I20260927 10:50:01.558547 11144 tablet_service.cc:1514] Processing DeleteTablet for tablet df245fefde974944bec876c3150b886d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:01 UTC) from {username='slave'} at 127.0.0.1:45644
I20260927 10:50:01.558712 11013 tablet_service.cc:1514] Processing DeleteTablet for tablet df245fefde974944bec876c3150b886d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:01 UTC) from {username='slave'} at 127.0.0.1:38462
I20260927 10:50:01.558734 11275 tablet_service.cc:1514] Processing DeleteTablet for tablet df245fefde974944bec876c3150b886d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:01 UTC) from {username='slave'} at 127.0.0.1:48974
I20260927 10:50:01.559231 11383 tablet_replica.cc:330] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: stopping tablet replica
I20260927 10:50:01.559540 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 10951
I20260927 10:50:01.560413 11383 raft_consensus.cc:2229] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:01.560715 11383 raft_consensus.cc:2258] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:01.562031 11383 ts_tablet_manager.cc:1905] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:01.563297 11381 tablet_replica.cc:330] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a: stopping tablet replica
I20260927 10:50:01.563562 11381 raft_consensus.cc:2229] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:50:01.563877 11381 raft_consensus.cc:2258] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:01.565315 11381 ts_tablet_manager.cc:1905] T df245fefde974944bec876c3150b886d P 37b45b19704e4cdd8197ee0e25f6257a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:50:01.568811 10876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:32875 (error 108)
I20260927 10:50:01.569634 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 11082
I20260927 10:50:01.569857 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.569559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:32875 (local address 127.0.0.1:38474)
0927 10:50:01.569682 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.569758 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:50:01.570894 10876 catalog_manager.cc:4584] TS 1bc2da5b644a486da43d7a5d3cd80824 (127.6.31.129:32875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df245fefde974944bec876c3150b886d: Network error: Client connection negotiation failed: client connection to 127.6.31.129:32875: connect: Connection refused (error 111)
I20260927 10:50:01.584265 11383 ts_tablet_manager.cc:1918] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:01.584405 11383 log.cc:1196] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/df245fefde974944bec876c3150b886d
I20260927 10:50:01.584693 11383 ts_tablet_manager.cc:1939] T df245fefde974944bec876c3150b886d P 7260323f0d594b71a8f8d77ea70b6e4b: Deleting consensus metadata
I20260927 10:50:01.588486 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 11213
I20260927 10:50:01.589049 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.588684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:35343 (local address 127.0.0.1:45666)
0927 10:50:01.588888 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.588943 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.130:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 10:50:01.608566 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 10860
2026-09-27T10:50:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1167 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 10:50:01.630148 6270 mini_oidc.cc:152] Starting JWKS server
I20260927 10:50:01.631898 6270 webserver.cc:460] Webserver started at https://127.0.0.1:38421/ using document root <none> and password file <none>
I20260927 10:50:01.632097 6270 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:50:01.632378 6270 webserver.cc:460] Webserver started at http://127.0.0.1:42329/ using document root <none> and password file <none>
2026-09-27T10:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:01Z Disabled control of system clock
I20260927 10:50:01.641870 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37315
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:37315
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38421/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/testchainca.cert with env {}
W20260927 10:50:01.758378 11390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:01.758662 11390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:01.758704 11390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:01.763389 11390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:01.763461 11390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:01.763476 11390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:01.763494 11390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:01.763509 11390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:50:01.763525 11390 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:50:01.763540 11390 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38421/jwks.json
I20260927 10:50:01.769104 11390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:37315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38421/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:01.770155 11390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:01.771135 11390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:01.779778 11395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:01.779986 11398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:01.779803 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:01.779934 11390 server_base.cc:1034] running on GCE node
I20260927 10:50:01.780606 11390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:01.781723 11390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:01.782909 11390 hybrid_clock.cc:648] HybridClock initialized: now 1790506201782897 us; error 24 us; skew 500 ppm
I20260927 10:50:01.784751 11390 webserver.cc:460] Webserver started at http://127.6.31.190:33263/ using document root <none> and password file <none>
I20260927 10:50:01.785323 11390 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:01.785419 11390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:01.785640 11390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:01.787774 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "3ca3753394764200952273f84e0fbbbb"
format_stamp: "Formatted at 2026-09-27 10:50:01 on dist-test-slave-c60x"
server_key: "7007868404bdea87fceda405e807420e"
server_key_iv: "e3544ee25fe57bb4803be56bcbc95138"
server_key_version: "encryptionkey@0"
I20260927 10:50:01.788333 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "3ca3753394764200952273f84e0fbbbb"
format_stamp: "Formatted at 2026-09-27 10:50:01 on dist-test-slave-c60x"
server_key: "7007868404bdea87fceda405e807420e"
server_key_iv: "e3544ee25fe57bb4803be56bcbc95138"
server_key_version: "encryptionkey@0"
I20260927 10:50:01.791914 11390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:50:01.794162 11404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:01.795351 11390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:01.795476 11390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "3ca3753394764200952273f84e0fbbbb"
format_stamp: "Formatted at 2026-09-27 10:50:01 on dist-test-slave-c60x"
server_key: "7007868404bdea87fceda405e807420e"
server_key_iv: "e3544ee25fe57bb4803be56bcbc95138"
server_key_version: "encryptionkey@0"
I20260927 10:50:01.795583 11390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:01.805542 11390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:01.806209 11390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:01.806403 11390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:01.813519 11390 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:37315
I20260927 10:50:01.813541 11456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:37315 every 8 connection(s)
I20260927 10:50:01.814472 11390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:01.817319 11457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:01.818302 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 11390
I20260927 10:50:01.818434 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:01.818702 6270 external_mini_cluster.cc:1428] Setting key 5a2dacae2e97c0add6c78e2fc22d6824
I20260927 10:50:01.823344 11457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb: Bootstrap starting.
I20260927 10:50:01.825997 11457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:01.826828 11457 log.cc:824] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:01.827953 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.820066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.6.31.190:37315)
0927 10:50:01.820654 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.820669 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.820709 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:50:01.821330 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.821345 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.821721 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.821974 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:01.822300 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:01.823270 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:01.824405 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:01.824847 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:01.824961 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:01.825832 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:01.825837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:01.825849 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:01.826314 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:01.826369 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:01.826390 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:01.826473 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:01.826628 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 10:50:01.826911 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":131,"threads_started":1}
I20260927 10:50:01.829111 11457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb: No bootstrap required, opened a new log
I20260927 10:50:01.830785 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:50:01.833138 11457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3753394764200952273f84e0fbbbb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37315 } }
I20260927 10:50:01.833387 11457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:01.833464 11457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca3753394764200952273f84e0fbbbb, State: Initialized, Role: FOLLOWER
I20260927 10:50:01.833930 11457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3753394764200952273f84e0fbbbb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37315 } }
I20260927 10:50:01.834075 11457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:01.834154 11457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:01.834251 11457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:01.835287 11457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3753394764200952273f84e0fbbbb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37315 } }
I20260927 10:50:01.835628 11457 leader_election.cc:302] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ca3753394764200952273f84e0fbbbb; no voters:
I20260927 10:50:01.835943 11457 leader_election.cc:288] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:01.836108 11462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:01.836361 11462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [term 1 LEADER]: Becoming Leader. State: Replica: 3ca3753394764200952273f84e0fbbbb, State: Running, Role: LEADER
I20260927 10:50:01.836690 11462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3753394764200952273f84e0fbbbb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37315 } }
I20260927 10:50:01.837327 11457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:01.838481 11463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ca3753394764200952273f84e0fbbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3753394764200952273f84e0fbbbb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37315 } } }
I20260927 10:50:01.838610 11463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:01.838878 11464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ca3753394764200952273f84e0fbbbb. Latest consensus state: current_term: 1 leader_uuid: "3ca3753394764200952273f84e0fbbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3753394764200952273f84e0fbbbb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37315 } } }
I20260927 10:50:01.838979 11464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:01.839049 11471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:01.842226 11471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:01.847760 11471 catalog_manager.cc:1347] Generated new cluster ID: abeaf261f09e45fb9e8c960dc64943c4
I20260927 10:50:01.847846 11471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:01.859714 11471 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:01.860673 11471 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:01.868958 11471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ca3753394764200952273f84e0fbbbb: Generated new TSK 0
I20260927 10:50:01.869769 11471 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:50:01.896761 11487 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 10:50:01.896889 11487 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:50:01.897464 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:01.879251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.6.31.190:37315)
0927 10:50:01.879380 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:01.879384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:01.879647 (+ 263us) server_negotiation.cc:407] Connection header received
0927 10:50:01.879843 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:01.879847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:01.879908 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:01.880019 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:50:01.880249 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:01.881096 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:01.882043 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:01.882410 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:01.882512 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:01.882909 (+ 397us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:50:01.897045 (+ 14136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:50:01.897320 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 10:50:01.897541 11460 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 10:50:01.898631 11486 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:01.879157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37315 (local address 127.0.0.1:57304)
0927 10:50:01.879546 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:01.879559 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:01.879672 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:01.880034 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:01.880036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:01.880047 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:50:01.880146 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:01.880150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:01.881235 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:01.881238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:01.881914 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:01.881923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:01.882562 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:01.882566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:01.882639 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:01.882700 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:50:01.897298 (+ 14598us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:50:01.897493 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.31.190:37315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":96,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.31.190:37315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 10:50:01.899622 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 11390
2026-09-27T10:50:01Z chronyd exiting
I20260927 10:50:01.937531 6270 test_util.cc:175] -----------------------------------------------
I20260927 10:50:01.937623 6270 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506175715859-6270-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 10:50:01.938705 6270 mini_oidc.cc:152] Starting JWKS server
I20260927 10:50:01.939713 6270 webserver.cc:460] Webserver started at https://127.0.0.1:40419/ using document root <none> and password file <none>
I20260927 10:50:01.939836 6270 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:50:01.940124 6270 webserver.cc:460] Webserver started at http://127.0.0.1:39181/ using document root <none> and password file <none>
2026-09-27T10:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:01Z Disabled control of system clock
I20260927 10:50:01.949918 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46137
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:46137
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40419/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/testchainca.cert with env {}
W20260927 10:50:02.066701 11493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:02.066955 11493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:02.066994 11493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:02.070871 11493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:02.070938 11493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:02.070953 11493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:02.070971 11493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:02.070986 11493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:50:02.071002 11493 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:50:02.071015 11493 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40419/jwks.json
I20260927 10:50:02.076565 11493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:46137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40419/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:02.077620 11493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:02.078609 11493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:02.087195 11499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:02.087229 11498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:02.087206 11501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:02.087441 11493 server_base.cc:1034] running on GCE node
I20260927 10:50:02.088088 11493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:02.089272 11493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:02.090466 11493 hybrid_clock.cc:648] HybridClock initialized: now 1790506202090441 us; error 46 us; skew 500 ppm
I20260927 10:50:02.092447 11493 webserver.cc:460] Webserver started at http://127.6.31.190:39137/ using document root <none> and password file <none>
I20260927 10:50:02.093086 11493 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:02.093237 11493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:02.093459 11493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:02.095544 11493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "ebfd2892391d438eaddecaf6c1ccaf7f"
format_stamp: "Formatted at 2026-09-27 10:50:02 on dist-test-slave-c60x"
server_key: "95dac89ee93d2b17d180a5ea1d192523"
server_key_iv: "3e015cb64cc2fff234d3feeef300e149"
server_key_version: "encryptionkey@0"
I20260927 10:50:02.096128 11493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "ebfd2892391d438eaddecaf6c1ccaf7f"
format_stamp: "Formatted at 2026-09-27 10:50:02 on dist-test-slave-c60x"
server_key: "95dac89ee93d2b17d180a5ea1d192523"
server_key_iv: "3e015cb64cc2fff234d3feeef300e149"
server_key_version: "encryptionkey@0"
I20260927 10:50:02.099665 11493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:02.101910 11507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:02.103191 11493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:02.103322 11493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "ebfd2892391d438eaddecaf6c1ccaf7f"
format_stamp: "Formatted at 2026-09-27 10:50:02 on dist-test-slave-c60x"
server_key: "95dac89ee93d2b17d180a5ea1d192523"
server_key_iv: "3e015cb64cc2fff234d3feeef300e149"
server_key_version: "encryptionkey@0"
I20260927 10:50:02.103431 11493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:02.114609 11493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:02.115289 11493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:02.115471 11493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:02.121937 11493 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:46137
I20260927 10:50:02.121944 11559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:46137 every 8 connection(s)
I20260927 10:50:02.122800 11493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:02.125833 11560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:02.126085 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 11493
I20260927 10:50:02.126240 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:02.126520 6270 external_mini_cluster.cc:1428] Setting key bff0e2b4c317013dfbaa8fc037330f09
I20260927 10:50:02.132431 11560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f: Bootstrap starting.
I20260927 10:50:02.134915 11560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:02.135535 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:02.127808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.6.31.190:46137)
0927 10:50:02.128396 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:02.128411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:02.128446 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:50:02.129095 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:02.129112 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:02.129483 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:02.129783 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:02.130101 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.131024 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.132232 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.132739 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.132852 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:02.133722 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:02.133727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:02.133738 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:02.134103 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:02.134152 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:02.134170 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:02.134260 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:02.134424 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 10:50:02.134711 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":146,"threads_started":1}
I20260927 10:50:02.135684 11560 log.cc:824] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:02.137647 11560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f: No bootstrap required, opened a new log
I20260927 10:50:02.138367 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:50:02.141461 11560 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd2892391d438eaddecaf6c1ccaf7f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46137 } }
I20260927 10:50:02.141719 11560 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:02.141800 11560 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfd2892391d438eaddecaf6c1ccaf7f, State: Initialized, Role: FOLLOWER
I20260927 10:50:02.142341 11560 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd2892391d438eaddecaf6c1ccaf7f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46137 } }
I20260927 10:50:02.142534 11560 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:02.142657 11560 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:02.142764 11560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:02.143764 11560 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd2892391d438eaddecaf6c1ccaf7f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46137 } }
I20260927 10:50:02.144143 11560 leader_election.cc:302] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebfd2892391d438eaddecaf6c1ccaf7f; no voters:
I20260927 10:50:02.144455 11560 leader_election.cc:288] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:02.144608 11565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:02.144850 11565 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [term 1 LEADER]: Becoming Leader. State: Replica: ebfd2892391d438eaddecaf6c1ccaf7f, State: Running, Role: LEADER
I20260927 10:50:02.145193 11565 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd2892391d438eaddecaf6c1ccaf7f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46137 } }
I20260927 10:50:02.145793 11560 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:02.146955 11567 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebfd2892391d438eaddecaf6c1ccaf7f. Latest consensus state: current_term: 1 leader_uuid: "ebfd2892391d438eaddecaf6c1ccaf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd2892391d438eaddecaf6c1ccaf7f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46137 } } }
I20260927 10:50:02.147111 11567 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:02.146939 11566 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebfd2892391d438eaddecaf6c1ccaf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd2892391d438eaddecaf6c1ccaf7f" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46137 } } }
I20260927 10:50:02.147250 11566 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:02.148146 11574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:02.150916 11574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:02.156246 11574 catalog_manager.cc:1347] Generated new cluster ID: 2da6ccfd2f2f457f84049739126ab3ef
I20260927 10:50:02.156312 11574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:02.164697 11574 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:02.165846 11574 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:02.176525 11574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebfd2892391d438eaddecaf6c1ccaf7f: Generated new TSK 0
I20260927 10:50:02.177340 11574 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:50:02.206873 11590 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 10:50:02.206962 11590 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:50:02.207748 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:02.191249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.6.31.190:46137)
0927 10:50:02.191387 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:02.191392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:02.191594 (+ 202us) server_negotiation.cc:407] Connection header received
0927 10:50:02.191757 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:02.191760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:02.191808 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:02.191898 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:50:02.192161 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.192694 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.193906 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.194121 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.194201 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:02.194506 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:50:02.207331 (+ 12825us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:50:02.207627 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 10:50:02.207826 11563 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 10:50:02.207978 11589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:02.191190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46137 (local address 127.0.0.1:39834)
0927 10:50:02.191498 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:02.191514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:02.191635 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:02.191927 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:02.191930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:02.191938 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:50:02.192060 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:02.192066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.192802 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.192806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:02.193793 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:02.193802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.194230 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.194234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:02.194326 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:02.194353 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:50:02.207550 (+ 13197us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:50:02.207744 (+ 194us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.31.190:46137: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
I20260927 10:50:02.208632 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 11493
2026-09-27T10:50:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 10:50:02.247972 6270 mini_oidc.cc:152] Starting JWKS server
I20260927 10:50:02.248824 6270 webserver.cc:460] Webserver started at https://127.0.0.1:36883/ using document root <none> and password file <none>
I20260927 10:50:02.248929 6270 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:50:02.249207 6270 webserver.cc:460] Webserver started at http://127.0.0.1:41103/ using document root <none> and password file <none>
2026-09-27T10:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:02Z Disabled control of system clock
I20260927 10:50:02.257736 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37037
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:37037
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36883/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:50:02.365209 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:02.365464 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:02.365502 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:02.369434 11596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:02.369498 11596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:02.369514 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:02.369530 11596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:02.369546 11596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:50:02.369562 11596 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:50:02.369575 11596 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36883/jwks.json
I20260927 10:50:02.374254 11596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:37037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36883/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:02.375409 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:02.376782 11596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:02.385263 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:02.385339 11596 server_base.cc:1034] running on GCE node
W20260927 10:50:02.385277 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:02.385284 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:02.385977 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:02.387246 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:02.388445 11596 hybrid_clock.cc:648] HybridClock initialized: now 1790506202388427 us; error 33 us; skew 500 ppm
I20260927 10:50:02.390290 11596 webserver.cc:460] Webserver started at http://127.6.31.190:32969/ using document root <none> and password file <none>
I20260927 10:50:02.390898 11596 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:02.390990 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:02.391268 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:02.393349 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "7cbf69541871443c8a4563c3cad39712"
format_stamp: "Formatted at 2026-09-27 10:50:02 on dist-test-slave-c60x"
server_key: "04e5a2d6a1cdcdac0df71f482bdecede"
server_key_iv: "24a62e8f705b47dfa3704cbc1fa1179f"
server_key_version: "encryptionkey@0"
I20260927 10:50:02.393920 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "7cbf69541871443c8a4563c3cad39712"
format_stamp: "Formatted at 2026-09-27 10:50:02 on dist-test-slave-c60x"
server_key: "04e5a2d6a1cdcdac0df71f482bdecede"
server_key_iv: "24a62e8f705b47dfa3704cbc1fa1179f"
server_key_version: "encryptionkey@0"
I20260927 10:50:02.397624 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:50:02.400014 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:02.401226 11596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:02.401362 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "7cbf69541871443c8a4563c3cad39712"
format_stamp: "Formatted at 2026-09-27 10:50:02 on dist-test-slave-c60x"
server_key: "04e5a2d6a1cdcdac0df71f482bdecede"
server_key_iv: "24a62e8f705b47dfa3704cbc1fa1179f"
server_key_version: "encryptionkey@0"
I20260927 10:50:02.401476 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:02.422289 11596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:02.423007 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:02.423244 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:02.429961 11596 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:37037
I20260927 10:50:02.429986 11662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:37037 every 8 connection(s)
I20260927 10:50:02.430882 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:02.432974 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 11596
I20260927 10:50:02.433090 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:02.433411 6270 external_mini_cluster.cc:1428] Setting key 2ecf88fc8be7e78627dd356201f4e4f4
I20260927 10:50:02.434767 11663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:02.441174 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712: Bootstrap starting.
I20260927 10:50:02.442453 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:02.434733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.6.31.190:37037)
0927 10:50:02.435389 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:02.435405 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:02.435440 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:50:02.436098 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:02.436113 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:02.436494 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:02.436770 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:02.437130 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.438080 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.439294 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.439765 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.439880 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:02.440753 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:02.440758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:02.440771 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:02.441158 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:02.441209 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:02.441227 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:02.441343 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:02.441486 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 10:50:02.441738 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
I20260927 10:50:02.443689 11663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:02.444590 11663 log.cc:824] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:02.445385 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:50:02.446841 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712: No bootstrap required, opened a new log
I20260927 10:50:02.450484 11663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbf69541871443c8a4563c3cad39712" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37037 } }
I20260927 10:50:02.450775 11663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:02.450830 11663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cbf69541871443c8a4563c3cad39712, State: Initialized, Role: FOLLOWER
I20260927 10:50:02.451297 11663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbf69541871443c8a4563c3cad39712" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37037 } }
I20260927 10:50:02.451422 11663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:02.451470 11663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:02.451541 11663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:02.452394 11663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbf69541871443c8a4563c3cad39712" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37037 } }
I20260927 10:50:02.452687 11663 leader_election.cc:302] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cbf69541871443c8a4563c3cad39712; no voters:
I20260927 10:50:02.452932 11663 leader_election.cc:288] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:02.453063 11668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:02.453310 11668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [term 1 LEADER]: Becoming Leader. State: Replica: 7cbf69541871443c8a4563c3cad39712, State: Running, Role: LEADER
I20260927 10:50:02.453625 11668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbf69541871443c8a4563c3cad39712" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37037 } }
I20260927 10:50:02.454198 11663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:02.455255 11669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cbf69541871443c8a4563c3cad39712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbf69541871443c8a4563c3cad39712" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37037 } } }
I20260927 10:50:02.455392 11669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:02.455677 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cbf69541871443c8a4563c3cad39712. Latest consensus state: current_term: 1 leader_uuid: "7cbf69541871443c8a4563c3cad39712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbf69541871443c8a4563c3cad39712" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37037 } } }
I20260927 10:50:02.455780 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:02.455857 11677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:02.458977 11677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:02.464566 11677 catalog_manager.cc:1347] Generated new cluster ID: 7413734f801b49508be74e3e5fcba20c
I20260927 10:50:02.464654 11677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:02.478482 11677 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:02.479683 11677 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:02.486438 11677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cbf69541871443c8a4563c3cad39712: Generated new TSK 0
I20260927 10:50:02.487272 11677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:50:02.503535 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:02.498318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.6.31.190:37037)
0927 10:50:02.498451 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:02.498457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:02.498598 (+ 141us) server_negotiation.cc:407] Connection header received
0927 10:50:02.498831 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:02.498834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:02.498886 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:02.498986 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:50:02.499286 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.499814 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.501047 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.501303 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.501383 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:02.501697 (+ 314us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:50:02.503178 (+ 1481us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:50:02.503395 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 10:50:02.503697 11666 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 10:50:02.506124 11692 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:02.498171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37037 (local address 127.0.0.1:37038)
0927 10:50:02.498497 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:02.498513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:02.498633 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:02.499014 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:02.499020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:02.499029 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:50:02.499171 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:02.499177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.499925 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.499929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:02.500918 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:02.500928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.501420 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.501424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:02.501512 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:02.501540 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:50:02.504093 (+ 2553us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:50:02.505865 (+ 1772us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.31.190:37037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":1508,"thread_start_us":112,"threads_started":1}
I20260927 10:50:02.506661 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 11596
E20260927 10:50:02.515645 11693 webserver.cc:533] Webserver: sslize failed:
E20260927 10:50:02.515718 11693 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T10:50:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T10:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:02Z Disabled control of system clock
I20260927 10:50:02.553799 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36379
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:36379
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/testchainca.cert with env {}
W20260927 10:50:02.661975 11697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:02.662253 11697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:02.662294 11697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:02.666334 11697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:02.666390 11697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:02.666405 11697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:02.666422 11697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:02.666437 11697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:50:02.666453 11697 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:50:02.666467 11697 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 10:50:02.671762 11697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:36379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:02.672758 11697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:02.673732 11697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:02.682327 11705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:02.682318 11702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:02.682318 11703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:02.682526 11697 server_base.cc:1034] running on GCE node
I20260927 10:50:02.683290 11697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:02.684525 11697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:02.685715 11697 hybrid_clock.cc:648] HybridClock initialized: now 1790506202685684 us; error 45 us; skew 500 ppm
I20260927 10:50:02.687661 11697 webserver.cc:460] Webserver started at http://127.6.31.190:34759/ using document root <none> and password file <none>
I20260927 10:50:02.688266 11697 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:02.688374 11697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:02.688602 11697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:02.690631 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "58a37fc031c04d84ad3ff51c633896e6"
format_stamp: "Formatted at 2026-09-27 10:50:02 on dist-test-slave-c60x"
server_key: "3e7ea9bcf3517827aa32963d2dd33eb1"
server_key_iv: "4e85b49e796377b011158b012014ede5"
server_key_version: "encryptionkey@0"
I20260927 10:50:02.691272 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "58a37fc031c04d84ad3ff51c633896e6"
format_stamp: "Formatted at 2026-09-27 10:50:02 on dist-test-slave-c60x"
server_key: "3e7ea9bcf3517827aa32963d2dd33eb1"
server_key_iv: "4e85b49e796377b011158b012014ede5"
server_key_version: "encryptionkey@0"
I20260927 10:50:02.694826 11697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:02.697150 11711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:02.698287 11697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:50:02.698424 11697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "58a37fc031c04d84ad3ff51c633896e6"
format_stamp: "Formatted at 2026-09-27 10:50:02 on dist-test-slave-c60x"
server_key: "3e7ea9bcf3517827aa32963d2dd33eb1"
server_key_iv: "4e85b49e796377b011158b012014ede5"
server_key_version: "encryptionkey@0"
I20260927 10:50:02.698531 11697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:02.717695 11697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:02.718391 11697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:02.718591 11697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:02.725468 11697 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:36379
I20260927 10:50:02.725482 11763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:36379 every 8 connection(s)
I20260927 10:50:02.726398 11697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:02.729156 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 11697
I20260927 10:50:02.729291 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:02.729583 11764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:02.729583 6270 external_mini_cluster.cc:1428] Setting key 14548396d97b520d8018bc1707f9149b
I20260927 10:50:02.735656 11764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6: Bootstrap starting.
I20260927 10:50:02.737936 11764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:02.738476 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:02.730978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.6.31.190:36379)
0927 10:50:02.731611 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:02.731639 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:02.731672 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:50:02.732265 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:02.732280 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:02.732674 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:02.733010 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:02.733406 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.734339 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.735265 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.735759 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.735936 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:02.736811 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:02.736816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:02.736828 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:02.737188 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:02.737239 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:02.737259 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:02.737364 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:02.737510 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 10:50:02.737744 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":123,"threads_started":1}
I20260927 10:50:02.738673 11764 log.cc:824] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:02.740684 11764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6: No bootstrap required, opened a new log
I20260927 10:50:02.741165 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:50:02.744318 11764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a37fc031c04d84ad3ff51c633896e6" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36379 } }
I20260927 10:50:02.744518 11764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:02.744596 11764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a37fc031c04d84ad3ff51c633896e6, State: Initialized, Role: FOLLOWER
I20260927 10:50:02.745007 11764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a37fc031c04d84ad3ff51c633896e6" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36379 } }
I20260927 10:50:02.745146 11764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:02.745209 11764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:02.745324 11764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:02.746199 11764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a37fc031c04d84ad3ff51c633896e6" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36379 } }
I20260927 10:50:02.746515 11764 leader_election.cc:302] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a37fc031c04d84ad3ff51c633896e6; no voters:
I20260927 10:50:02.746801 11764 leader_election.cc:288] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:02.746946 11769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:02.747191 11769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [term 1 LEADER]: Becoming Leader. State: Replica: 58a37fc031c04d84ad3ff51c633896e6, State: Running, Role: LEADER
I20260927 10:50:02.747491 11769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a37fc031c04d84ad3ff51c633896e6" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36379 } }
I20260927 10:50:02.748119 11764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:02.749197 11771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a37fc031c04d84ad3ff51c633896e6. Latest consensus state: current_term: 1 leader_uuid: "58a37fc031c04d84ad3ff51c633896e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a37fc031c04d84ad3ff51c633896e6" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36379 } } }
I20260927 10:50:02.749336 11771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:02.749588 11770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a37fc031c04d84ad3ff51c633896e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a37fc031c04d84ad3ff51c633896e6" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36379 } } }
I20260927 10:50:02.749687 11770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:02.749806 11778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:02.752926 11778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:02.757750 11778 catalog_manager.cc:1347] Generated new cluster ID: ea57a8f55f9843a2b446333697d34c7d
I20260927 10:50:02.757817 11778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:02.767864 11778 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:02.768781 11778 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:02.775591 11778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a37fc031c04d84ad3ff51c633896e6: Generated new TSK 0
I20260927 10:50:02.776301 11778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:50:02.899701 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:02.784227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.6.31.190:36379)
0927 10:50:02.784343 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:02.784348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:02.784499 (+ 151us) server_negotiation.cc:407] Connection header received
0927 10:50:02.784656 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:02.784660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:02.784716 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:02.784836 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:50:02.785089 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.785906 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.786746 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.787106 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.787207 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:02.787482 (+ 275us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:50:02.899294 (+111812us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:50:02.899571 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50386: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 10:50:02.899775 11767 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50386: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 10:50:02.900000 11793 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:02.784139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36379 (local address 127.0.0.1:50386)
0927 10:50:02.784403 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:02.784416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:02.784513 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:02.784865 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:02.784868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:02.784876 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:50:02.784969 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:02.784974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.786018 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.786021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:02.786626 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:02.786634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:02.787220 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:02.787223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:02.787295 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:02.787317 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:50:02.899557 (+112240us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:50:02.899792 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.31.190:36379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260927 10:50:02.900653 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 11697
2026-09-27T10:50:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (374 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (128 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:50:03 dist-test-slave-c60x krb5kdc[11805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:50:03 dist-test-slave-c60x krb5kdc[11805](info): set up 2 sockets
Sep 27 10:50:03 dist-test-slave-c60x krb5kdc[11805](info): commencing operation
krb5kdc: starting...
W20260927 10:50:04.191903 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:50:04 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:50:04.313759 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38545
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:38545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:04.421243 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:04.421536 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:04.421577 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:04.425448 11818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:04.425499 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:04.425518 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:04.425535 11818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:04.425550 11818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:04.430653 11818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:38545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:04.431613 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:04.432538 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:04.440742 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:04.440734 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:04.440847 11818 server_base.cc:1034] running on GCE node
W20260927 10:50:04.440734 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:04.441599 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:04.442617 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:04.443831 11818 hybrid_clock.cc:648] HybridClock initialized: now 1790506204443808 us; error 42 us; skew 500 ppm
Sep 27 10:50:04 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506204, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:04.451653 11818 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:50:04.452598 11818 webserver.cc:460] Webserver started at http://127.6.31.190:44145/ using document root <none> and password file <none>
I20260927 10:50:04.453162 11818 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:04.453251 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:04.453467 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:04.455547 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "194aa95c616540cd9d55ad5aa1be74a9"
format_stamp: "Formatted at 2026-09-27 10:50:04 on dist-test-slave-c60x"
server_key: "60d602d369a888657b4c503a2a5cc621"
server_key_iv: "f4f7d14f235370c3b42a9b34d798754b"
server_key_version: "encryptionkey@0"
I20260927 10:50:04.456113 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "194aa95c616540cd9d55ad5aa1be74a9"
format_stamp: "Formatted at 2026-09-27 10:50:04 on dist-test-slave-c60x"
server_key: "60d602d369a888657b4c503a2a5cc621"
server_key_iv: "f4f7d14f235370c3b42a9b34d798754b"
server_key_version: "encryptionkey@0"
I20260927 10:50:04.459769 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:04.462008 11833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:04.463379 11818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:04.463514 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "194aa95c616540cd9d55ad5aa1be74a9"
format_stamp: "Formatted at 2026-09-27 10:50:04 on dist-test-slave-c60x"
server_key: "60d602d369a888657b4c503a2a5cc621"
server_key_iv: "f4f7d14f235370c3b42a9b34d798754b"
server_key_version: "encryptionkey@0"
I20260927 10:50:04.463627 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:04.494866 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:04.498144 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:04.498343 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:04.505566 11818 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:38545
I20260927 10:50:04.505586 11885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:38545 every 8 connection(s)
I20260927 10:50:04.506520 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:04.509274 11886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:04.510108 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 11818
I20260927 10:50:04.510253 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:04.510530 6270 external_mini_cluster.cc:1428] Setting key 4afc28f94382a24f51667a100076ec0b
I20260927 10:50:04.516258 11886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9: Bootstrap starting.
I20260927 10:50:04.518776 11886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:04.519724 11886 log.cc:824] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:50:04 dist-test-slave-c60x krb5kdc[11805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:50:04.521795 11886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9: No bootstrap required, opened a new log
I20260927 10:50:04.526484 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:04.512398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.6.31.190:38545)
0927 10:50:04.512947 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:04.512967 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:04.513021 (+ 54us) server_negotiation.cc:407] Connection header received
0927 10:50:04.514006 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:04.514032 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:04.514497 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:04.514820 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:04.515722 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:04.516814 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:04.517947 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:04.518428 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:04.518560 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:04.521371 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:04.521377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:04.521392 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:04.521420 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:04.523397 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:04.523809 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:04.523815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:04.523821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:04.523960 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:04.524296 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:04.524301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:04.524303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:04.524812 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:04.525000 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:04.525261 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 10:50:04.525548 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":143,"threads_started":1}
I20260927 10:50:04.526445 11886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194aa95c616540cd9d55ad5aa1be74a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38545 } }
I20260927 10:50:04.526712 11886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:04.526767 11886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194aa95c616540cd9d55ad5aa1be74a9, State: Initialized, Role: FOLLOWER
I20260927 10:50:04.527241 11886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194aa95c616540cd9d55ad5aa1be74a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38545 } }
I20260927 10:50:04.527384 11886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:04.527449 11886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:04.527539 11886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:04.528365 11886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194aa95c616540cd9d55ad5aa1be74a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38545 } }
I20260927 10:50:04.528735 11886 leader_election.cc:302] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 194aa95c616540cd9d55ad5aa1be74a9; no voters:
I20260927 10:50:04.529016 11886 leader_election.cc:288] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:04.529116 11891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:04.529386 11891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [term 1 LEADER]: Becoming Leader. State: Replica: 194aa95c616540cd9d55ad5aa1be74a9, State: Running, Role: LEADER
I20260927 10:50:04.529711 11891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194aa95c616540cd9d55ad5aa1be74a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38545 } }
I20260927 10:50:04.530267 11886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:04.531404 11893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 194aa95c616540cd9d55ad5aa1be74a9. Latest consensus state: current_term: 1 leader_uuid: "194aa95c616540cd9d55ad5aa1be74a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194aa95c616540cd9d55ad5aa1be74a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38545 } } }
I20260927 10:50:04.531406 11892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "194aa95c616540cd9d55ad5aa1be74a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194aa95c616540cd9d55ad5aa1be74a9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38545 } } }
I20260927 10:50:04.531548 11892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:04.531905 11900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:04.531548 11893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:04.534849 11900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:04.540046 11900 catalog_manager.cc:1347] Generated new cluster ID: 5d874887fe7e49538d9a45d54afb40df
I20260927 10:50:04.540141 11900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:04.548311 11900 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:04.549253 11900 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:04.555162 11900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 194aa95c616540cd9d55ad5aa1be74a9: Generated new TSK 0
I20260927 10:50:04.555779 11900 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:50:04.590725 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38545
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:04.709584 11912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:04.709890 11912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:04.709962 11912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:04.714051 11912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:04.714146 11912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:04.714236 11912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:04.719810 11912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:04.720894 11912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:04.721925 11912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:04.731386 11920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:04.731419 11918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:04.731418 11917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:04.731401 11912 server_base.cc:1034] running on GCE node
I20260927 10:50:04.732054 11912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:04.732693 11912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:04.733888 11912 hybrid_clock.cc:648] HybridClock initialized: now 1790506204733874 us; error 31 us; skew 500 ppm
Sep 27 10:50:04 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506204, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:04.736595 11912 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:50:04.737421 11912 webserver.cc:460] Webserver started at http://127.6.31.129:37621/ using document root <none> and password file <none>
I20260927 10:50:04.737967 11912 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:04.738050 11912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:04.738271 11912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:04.740320 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "ae5ef2b1acc348a5a8972c5173a37ec5"
format_stamp: "Formatted at 2026-09-27 10:50:04 on dist-test-slave-c60x"
server_key: "2ed6f0590b3a3ebe7b7f48f00fa9ff1e"
server_key_iv: "d3acd9e6c52d8103bb176354dbf4255f"
server_key_version: "encryptionkey@0"
I20260927 10:50:04.740890 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "ae5ef2b1acc348a5a8972c5173a37ec5"
format_stamp: "Formatted at 2026-09-27 10:50:04 on dist-test-slave-c60x"
server_key: "2ed6f0590b3a3ebe7b7f48f00fa9ff1e"
server_key_iv: "d3acd9e6c52d8103bb176354dbf4255f"
server_key_version: "encryptionkey@0"
I20260927 10:50:04.744556 11912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:04.746959 11927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:04.748399 11912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:50:04.748545 11912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "ae5ef2b1acc348a5a8972c5173a37ec5"
format_stamp: "Formatted at 2026-09-27 10:50:04 on dist-test-slave-c60x"
server_key: "2ed6f0590b3a3ebe7b7f48f00fa9ff1e"
server_key_iv: "d3acd9e6c52d8103bb176354dbf4255f"
server_key_version: "encryptionkey@0"
I20260927 10:50:04.748667 11912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:04.761852 11912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:04.764940 11912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:04.765143 11912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:04.765740 11912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:04.766621 11912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:04.766701 11912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:04.766769 11912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:04.766812 11912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:04.775710 11912 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:40359
I20260927 10:50:04.775727 12040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:40359 every 8 connection(s)
I20260927 10:50:04.776713 11912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:04.778475 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 11912
I20260927 10:50:04.778597 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:50:04.778918 6270 external_mini_cluster.cc:1428] Setting key 04fcda7321101494515562da2583d534
Sep 27 10:50:04 dist-test-slave-c60x krb5kdc[11805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506204, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:50:04.793314 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:04.778699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:37627 (local address 127.6.31.190:38545)
0927 10:50:04.778934 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:04.778937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:04.779911 (+ 974us) server_negotiation.cc:407] Connection header received
0927 10:50:04.781829 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:04.781831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:04.781878 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:04.781947 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:04.783356 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:04.784247 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:04.785736 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:04.786184 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:04.786329 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:04.789195 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:04.789200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:04.789204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:04.789237 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:04.790665 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:04.791349 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:04.791353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:04.791355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:04.791478 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:04.791878 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:04.791881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:04.791883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:04.792152 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:04.792256 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:04.792848 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 10:50:04.793003 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260927 10:50:04.794193 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:04.779213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38545 (local address 127.6.31.129:37627)
0927 10:50:04.779749 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:04.779803 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:04.781567 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:04.782139 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:04.782172 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:04.782634 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:04.783205 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:04.783220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:04.784394 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:04.784398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:04.785593 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:04.785604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:04.786364 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:04.786367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:04.786518 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:04.787253 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:04.787270 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:04.789000 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:04.790810 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:04.790817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:04.790854 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:04.791239 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:04.791572 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:04.791579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:04.791581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:04.791762 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:04.792276 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:04.792283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:04.792586 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 10:50:04.792755 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 10:50:04.793030 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":156,"threads_started":1}
I20260927 10:50:04.795469 12041 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38545
I20260927 10:50:04.795729 12041 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:04.796248 12041 heartbeater.cc:511] Master 127.6.31.190:38545 requested a full tablet report, sending...
I20260927 10:50:04.797808 11850 ts_manager.cc:194] Registered new tserver with Master: ae5ef2b1acc348a5a8972c5173a37ec5 (127.6.31.129:40359)
I20260927 10:50:04.798918 11850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:37627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:50:04.811457 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38545
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:04.922217 12046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:04.922478 12046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:04.922523 12046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:04.926491 12046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:04.926561 12046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:04.926620 12046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:04.932209 12046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:04.933259 12046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:04.934265 12046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:04.943981 12054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:04.944090 12051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:04.943990 12052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:04.944147 12046 server_base.cc:1034] running on GCE node
I20260927 10:50:04.944629 12046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:04.945325 12046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:04.946512 12046 hybrid_clock.cc:648] HybridClock initialized: now 1790506204946499 us; error 31 us; skew 500 ppm
Sep 27 10:50:04 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506204, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:04.949612 12046 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:50:04.950469 12046 webserver.cc:460] Webserver started at http://127.6.31.130:40327/ using document root <none> and password file <none>
I20260927 10:50:04.951030 12046 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:04.951139 12046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:04.951362 12046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:04.953516 12046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "b3399752744742ef86539f7bb81f9d55"
format_stamp: "Formatted at 2026-09-27 10:50:04 on dist-test-slave-c60x"
server_key: "fd71bcbe089a97b1b5b057f474e5dfa4"
server_key_iv: "05a8dfa76e9f29b933b8b0c8be50cf0b"
server_key_version: "encryptionkey@0"
I20260927 10:50:04.954061 12046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "b3399752744742ef86539f7bb81f9d55"
format_stamp: "Formatted at 2026-09-27 10:50:04 on dist-test-slave-c60x"
server_key: "fd71bcbe089a97b1b5b057f474e5dfa4"
server_key_iv: "05a8dfa76e9f29b933b8b0c8be50cf0b"
server_key_version: "encryptionkey@0"
I20260927 10:50:04.957616 12046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:50:04.960311 12061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:04.961621 12046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:04.961740 12046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "b3399752744742ef86539f7bb81f9d55"
format_stamp: "Formatted at 2026-09-27 10:50:04 on dist-test-slave-c60x"
server_key: "fd71bcbe089a97b1b5b057f474e5dfa4"
server_key_iv: "05a8dfa76e9f29b933b8b0c8be50cf0b"
server_key_version: "encryptionkey@0"
I20260927 10:50:04.961850 12046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:04.989537 12046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:04.992432 12046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:04.992625 12046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:04.993206 12046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:04.993984 12046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:04.994052 12046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:04.994120 12046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:04.994167 12046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:05.002888 12046 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:45591
I20260927 10:50:05.002910 12174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:45591 every 8 connection(s)
I20260927 10:50:05.003785 12046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:05.008201 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 12046
I20260927 10:50:05.008344 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:50:05.008591 6270 external_mini_cluster.cc:1428] Setting key d75b969422b0bd9b9f9a7dde5ecff58e
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506204, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:50:05.018736 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:05.005520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:49747 (local address 127.6.31.190:38545)
0927 10:50:05.005646 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:05.005649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:05.006443 (+ 794us) server_negotiation.cc:407] Connection header received
0927 10:50:05.007808 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:05.007811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:05.007856 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:05.007927 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:05.009163 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.009872 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.011732 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.012120 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.012256 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:05.014757 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:05.014765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:05.014767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:05.014790 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:05.016650 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.017120 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.017126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.017129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.017225 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.017575 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.017580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.017585 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.017835 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:05.017949 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:05.018393 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 10:50:05.018516 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:50:05.019513 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:05.005882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38545 (local address 127.6.31.130:49747)
0927 10:50:05.006303 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:05.006343 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:05.007606 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:05.008076 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:05.008082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:05.008491 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:05.009007 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:05.009019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.010003 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.010006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:05.011574 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:05.011587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.012284 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.012286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:05.012418 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:05.012993 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:05.013009 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:05.014593 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:05.016771 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:05.016776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:05.016786 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:05.017021 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:05.017326 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:05.017328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:05.017330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:05.017483 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:05.017936 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:05.017941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:05.018159 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 10:50:05.018310 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:50:05.018544 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260927 10:50:05.020709 12175 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38545
I20260927 10:50:05.020967 12175 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:05.021489 12175 heartbeater.cc:511] Master 127.6.31.190:38545 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
I20260927 10:50:05.022528 11850 ts_manager.cc:194] Registered new tserver with Master: b3399752744742ef86539f7bb81f9d55 (127.6.31.130:45591)
I20260927 10:50:05.023138 11850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:49747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:50:05.039008 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38545
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:05.150574 12180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:05.150921 12180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:05.150967 12180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:05.155223 12180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:05.155273 12180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:05.155332 12180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:05.160531 12180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:05.161559 12180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:05.162550 12180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:05.172262 12186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:05.172345 12180 server_base.cc:1034] running on GCE node
W20260927 10:50:05.172299 12185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:05.172273 12188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:05.172822 12180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:05.173452 12180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:05.174605 12180 hybrid_clock.cc:648] HybridClock initialized: now 1790506205174593 us; error 26 us; skew 500 ppm
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:05.177366 12180 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:50:05.178197 12180 webserver.cc:460] Webserver started at http://127.6.31.131:33707/ using document root <none> and password file <none>
I20260927 10:50:05.178755 12180 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:05.178815 12180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:05.179018 12180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:05.181110 12180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "794081dd4fdd4a0d81d316321054a1bc"
format_stamp: "Formatted at 2026-09-27 10:50:05 on dist-test-slave-c60x"
server_key: "7d968005c2b32c3b0ff27fd99cebe76e"
server_key_iv: "c9f8e06f2aa855cec0606daf4c2cb288"
server_key_version: "encryptionkey@0"
I20260927 10:50:05.181702 12180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "794081dd4fdd4a0d81d316321054a1bc"
format_stamp: "Formatted at 2026-09-27 10:50:05 on dist-test-slave-c60x"
server_key: "7d968005c2b32c3b0ff27fd99cebe76e"
server_key_iv: "c9f8e06f2aa855cec0606daf4c2cb288"
server_key_version: "encryptionkey@0"
I20260927 10:50:05.185451 12180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:50:05.187896 12195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:05.189116 12180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:50:05.189230 12180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "794081dd4fdd4a0d81d316321054a1bc"
format_stamp: "Formatted at 2026-09-27 10:50:05 on dist-test-slave-c60x"
server_key: "7d968005c2b32c3b0ff27fd99cebe76e"
server_key_iv: "c9f8e06f2aa855cec0606daf4c2cb288"
server_key_version: "encryptionkey@0"
I20260927 10:50:05.189335 12180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:05.215790 12180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:05.219148 12180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:05.219374 12180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:05.220033 12180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:05.220903 12180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:05.220973 12180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:05.221048 12180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:05.221091 12180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:05.229724 12180 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:42623
I20260927 10:50:05.229764 12308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:42623 every 8 connection(s)
I20260927 10:50:05.230646 12180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:50:05.235435 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 12180
I20260927 10:50:05.235561 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:50:05.235829 6270 external_mini_cluster.cc:1428] Setting key 57bcaa2fe899061125d855f3b6c1cd44
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:50:05.244023 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:05.232644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:60729 (local address 127.6.31.190:38545)
0927 10:50:05.232785 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:05.232788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:05.233664 (+ 876us) server_negotiation.cc:407] Connection header received
0927 10:50:05.234998 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:05.235001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:05.235047 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:05.235182 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:05.236484 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.237213 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.238489 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.238704 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.238784 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:05.241029 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:05.241032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:05.241034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:05.241055 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:05.242303 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.242737 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.242739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.242741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.242808 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.243068 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.243071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.243072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.243301 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:05.243378 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:05.243820 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 10:50:05.243908 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:50:05.244796 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:05.233034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38545 (local address 127.6.31.131:60729)
0927 10:50:05.233495 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:05.233537 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:05.234798 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:05.235340 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:05.235346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:05.235824 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:05.236337 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:05.236351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.237321 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.237324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:05.238388 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:05.238397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.238822 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.238825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:05.238958 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:05.239513 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:05.239529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:05.240892 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:05.242398 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:05.242402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:05.242413 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:05.242660 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:05.242878 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:05.242880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:05.242881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:05.242999 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:05.243385 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:05.243390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:05.243611 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 10:50:05.243774 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 10:50:05.244006 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260927 10:50:05.245913 12309 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38545
I20260927 10:50:05.246107 12309 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:05.246557 12309 heartbeater.cc:511] Master 127.6.31.190:38545 requested a full tablet report, sending...
I20260927 10:50:05.247498 11850 ts_manager.cc:194] Registered new tserver with Master: 794081dd4fdd4a0d81d316321054a1bc (127.6.31.131:42623)
I20260927 10:50:05.248097 11850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:60729
I20260927 10:50:05.249464 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/test-admin.keytab.
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:05.268137 6270 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 10:50:05.270821 12316 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:05.269506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.6.31.129:40359)
0927 10:50:05.269798 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:05.269813 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:05.269836 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:50:05.269885 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:05.269889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:05.270067 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:05.270185 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:05.270628 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56366: BlockingRecv error: recv got EOF from 127.0.0.1:56366 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
W20260927 10:50:05.271231 6270 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5cc)
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:05.272231 6270 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:05.273111 6270 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:50:05.282310 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:05.273987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.6.31.129:40359)
0927 10:50:05.274111 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:05.274114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:05.274129 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:50:05.274248 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:05.274250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:05.274285 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:05.274348 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:05.274835 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.275702 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.276475 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.276838 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.276919 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:05.279215 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:05.279223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:05.279234 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:05.279255 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:05.280672 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.281094 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.281099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.281100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.281204 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.281476 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.281478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.281479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.281691 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:05.281841 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:05.282039 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 10:50:05.282186 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 10:50:05.286026 12316 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:05.285122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.6.31.129:40359)
0927 10:50:05.285229 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:05.285234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:05.285295 (+ 61us) server_negotiation.cc:407] Connection header received
0927 10:50:05.285427 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:05.285430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:05.285468 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:05.285547 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:05.285918 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56382: BlockingRecv error: recv got EOF from 127.0.0.1:56382 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 10:50:05.286260 6270 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:05.287298 6270 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:05.288149 6270 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:50:05.296381 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:05.288852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.6.31.129:40359)
0927 10:50:05.288954 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:05.288957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:05.289168 (+ 211us) server_negotiation.cc:407] Connection header received
0927 10:50:05.289225 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:05.289227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:05.289258 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:05.289318 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:05.289763 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.290288 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.291182 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.291390 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.291467 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:05.293562 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:05.293565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:05.293567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:05.293585 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:05.294888 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.295396 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.295399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.295400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.295468 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.295665 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.295667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.295668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.295799 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:05.295884 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:05.296069 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 10:50:05.296208 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 10:50:05.299117 12316 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:05.298212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.6.31.129:40359)
0927 10:50:05.298322 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:05.298326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:05.298455 (+ 129us) server_negotiation.cc:407] Connection header received
0927 10:50:05.298544 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:05.298547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:05.298579 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:05.298643 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:05.298994 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56406: BlockingRecv error: recv got EOF from 127.0.0.1:56406 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 10:50:05.299239 6270 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506175715859-6270-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:05.300277 6270 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:05.301193 6270 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[11805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:50:05.309369 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:05.302002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.6.31.129:40359)
0927 10:50:05.302110 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:05.302116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:05.302221 (+ 105us) server_negotiation.cc:407] Connection header received
0927 10:50:05.302308 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:05.302310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:05.302344 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:05.302407 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:05.302844 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.303398 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.304142 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:05.304343 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:05.304416 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:05.306541 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:05.306544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:05.306546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:05.306567 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:05.307984 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.308350 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.308353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.308354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.308420 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:05.308628 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:05.308640 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:05.308641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:05.308789 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:05.308879 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:05.309073 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 10:50:05.309160 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:50:05.310551 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 11912
I20260927 10:50:05.316457 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 12046
I20260927 10:50:05.321877 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 12180
I20260927 10:50:05.327128 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 11818
2026-09-27T10:50:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2180 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[12329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[12329](info): set up 2 sockets
Sep 27 10:50:05 dist-test-slave-c60x krb5kdc[12329](info): commencing operation
krb5kdc: starting...
W20260927 10:50:06.371548 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:50:06 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.190.keytab.
Entry for principal oryx/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.190.keytab.
I20260927 10:50:06.494194 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36259
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.190.keytab
--principal=oryx/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:06.617624 12342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:06.617871 12342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:06.617911 12342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:06.622053 12342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:06.622120 12342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:06.622139 12342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:06.622176 12342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:06.622192 12342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:06.627532 12342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:36259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.190.keytab
--principal=oryx/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:06.628559 12342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:06.629549 12342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:50:06.637846 12342 server_base.cc:1034] running on GCE node
W20260927 10:50:06.637812 12347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:06.637812 12348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:06.637825 12350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:06.638624 12342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:06.639762 12342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:06.640934 12342 hybrid_clock.cc:648] HybridClock initialized: now 1790506206640922 us; error 24 us; skew 500 ppm
Sep 27 10:50:06 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506206, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:06.643602 12342 init.cc:377] Logged in from keytab as oryx/127.6.31.190@KRBTEST.COM (short username oryx)
I20260927 10:50:06.644418 12342 webserver.cc:460] Webserver started at http://127.6.31.190:43673/ using document root <none> and password file <none>
I20260927 10:50:06.644945 12342 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:06.645028 12342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:06.645241 12342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:06.647244 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "1331601e10ce4de1915f1b5b26522952"
format_stamp: "Formatted at 2026-09-27 10:50:06 on dist-test-slave-c60x"
server_key: "7089ddeace23d3248c88265ec9cd1ef9"
server_key_iv: "c804afc5db3990b5f7969c3989117303"
server_key_version: "encryptionkey@0"
I20260927 10:50:06.647789 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "1331601e10ce4de1915f1b5b26522952"
format_stamp: "Formatted at 2026-09-27 10:50:06 on dist-test-slave-c60x"
server_key: "7089ddeace23d3248c88265ec9cd1ef9"
server_key_iv: "c804afc5db3990b5f7969c3989117303"
server_key_version: "encryptionkey@0"
I20260927 10:50:06.651434 12342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:06.653625 12357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:06.654764 12342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:50:06.654902 12342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "1331601e10ce4de1915f1b5b26522952"
format_stamp: "Formatted at 2026-09-27 10:50:06 on dist-test-slave-c60x"
server_key: "7089ddeace23d3248c88265ec9cd1ef9"
server_key_iv: "c804afc5db3990b5f7969c3989117303"
server_key_version: "encryptionkey@0"
I20260927 10:50:06.655009 12342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:06.670997 12342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:06.673790 12342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:06.673990 12342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:06.680768 12342 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:36259
I20260927 10:50:06.680799 12409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:36259 every 8 connection(s)
I20260927 10:50:06.681689 12342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:06.684289 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:06.689424 12410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: Bootstrap starting.
I20260927 10:50:06.690464 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 12342
I20260927 10:50:06.690588 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:06.690841 6270 external_mini_cluster.cc:1428] Setting key 5aa3f7c0e409f90ea6a20c74e3e734d3
I20260927 10:50:06.692085 12410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:06.692778 12410 log.cc:824] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:06.694855 12410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: No bootstrap required, opened a new log
I20260927 10:50:06.699296 12410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } }
I20260927 10:50:06.699529 12410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:06.699611 12410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1331601e10ce4de1915f1b5b26522952, State: Initialized, Role: FOLLOWER
Sep 27 10:50:06 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:06.700114 12410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } }
I20260927 10:50:06.700279 12410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:06.700347 12410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:06.700444 12410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:06.701341 12410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } }
I20260927 10:50:06.701721 12410 leader_election.cc:302] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1331601e10ce4de1915f1b5b26522952; no voters:
I20260927 10:50:06.702055 12410 leader_election.cc:288] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:06.702169 12415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:06.702401 12415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 1 LEADER]: Becoming Leader. State: Replica: 1331601e10ce4de1915f1b5b26522952, State: Running, Role: LEADER
I20260927 10:50:06.702701 12415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } }
I20260927 10:50:06.703473 12410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:06.704550 12416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } }
I20260927 10:50:06.704586 12417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1331601e10ce4de1915f1b5b26522952. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } }
I20260927 10:50:06.704702 12417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:06.704702 12416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:06.705245 12424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:06.706506 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:06.691944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.6.31.190:36259)
0927 10:50:06.692373 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:06.692390 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:06.692422 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:50:06.693076 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:06.693092 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:06.693500 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:06.693772 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:06.694646 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:06.695714 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:06.696885 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:06.697337 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:06.697446 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:06.700371 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:06.700378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:06.700416 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:06.700451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:06.702559 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:06.703060 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:06.703066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:06.703073 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:06.703252 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:06.703553 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:06.703557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:06.703559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:06.704088 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:06.704292 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:06.705225 (+ 933us) server_negotiation.cc:299] Negotiation successful
0927 10:50:06.705534 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
I20260927 10:50:06.708388 12424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:06.713508 12424 catalog_manager.cc:1347] Generated new cluster ID: 78e040796fe641b3a1d79c4740cdb9b0
I20260927 10:50:06.713574 12424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:06.726778 12424 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:06.727741 12424 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:06.732566 12424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: Generated new TSK 0
I20260927 10:50:06.733328 12424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab.
Entry for principal oryx/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab.
I20260927 10:50:06.769646 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36259
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab
--principal=oryx/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:06.877895 12436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:06.878206 12436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:06.878280 12436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:06.882298 12436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:06.882397 12436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:06.882491 12436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:06.887892 12436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab
--principal=oryx/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:06.888962 12436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:06.889962 12436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:50:06.899523 12436 server_base.cc:1034] running on GCE node
W20260927 10:50:06.899488 12441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:06.899488 12442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:06.899490 12444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:06.900142 12436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:06.900718 12436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:06.901897 12436 hybrid_clock.cc:648] HybridClock initialized: now 1790506206901884 us; error 25 us; skew 500 ppm
Sep 27 10:50:06 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506206, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:06.904698 12436 init.cc:377] Logged in from keytab as oryx/127.6.31.129@KRBTEST.COM (short username oryx)
I20260927 10:50:06.905615 12436 webserver.cc:460] Webserver started at http://127.6.31.129:42201/ using document root <none> and password file <none>
I20260927 10:50:06.906176 12436 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:06.906280 12436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:06.906514 12436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:06.908622 12436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "5acccd0f20d448e7b4414693d2073ca9"
format_stamp: "Formatted at 2026-09-27 10:50:06 on dist-test-slave-c60x"
server_key: "2b9a7219303d440c245cd6592e89a138"
server_key_iv: "2e7fb659dcb289c026f072f9a3fe2e8a"
server_key_version: "encryptionkey@0"
I20260927 10:50:06.909195 12436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "5acccd0f20d448e7b4414693d2073ca9"
format_stamp: "Formatted at 2026-09-27 10:50:06 on dist-test-slave-c60x"
server_key: "2b9a7219303d440c245cd6592e89a138"
server_key_iv: "2e7fb659dcb289c026f072f9a3fe2e8a"
server_key_version: "encryptionkey@0"
I20260927 10:50:06.913018 12436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:50:06.915563 12451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:06.916824 12436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:06.916965 12436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "5acccd0f20d448e7b4414693d2073ca9"
format_stamp: "Formatted at 2026-09-27 10:50:06 on dist-test-slave-c60x"
server_key: "2b9a7219303d440c245cd6592e89a138"
server_key_iv: "2e7fb659dcb289c026f072f9a3fe2e8a"
server_key_version: "encryptionkey@0"
I20260927 10:50:06.917076 12436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:06.930928 12436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:06.934195 12436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:06.934412 12436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:06.935055 12436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:06.935943 12436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:06.936033 12436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:06.936127 12436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:06.936173 12436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:06.944756 12436 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:36455
I20260927 10:50:06.944772 12564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:36455 every 8 connection(s)
I20260927 10:50:06.945652 12436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:06.945945 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 12436
I20260927 10:50:06.946062 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:50:06.946455 6270 external_mini_cluster.cc:1428] Setting key 01b058331a176e260e76fc7304a38b12
WARNING: no policy specified for oryx/127.6.31.130@KRBTEST.COM; defaulting to no policy
Sep 27 10:50:06 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506206, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.130@KRBTEST.COM" created.
I20260927 10:50:06.960431 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:06.947480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:49661 (local address 127.6.31.190:36259)
0927 10:50:06.947623 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:06.947627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:06.949070 (+ 1443us) server_negotiation.cc:407] Connection header received
0927 10:50:06.950450 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:06.950453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:06.950508 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:06.950586 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:06.951823 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:06.952413 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:06.953513 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:06.953751 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:06.953861 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:06.956321 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:06.956324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:06.956327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:06.956352 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:06.958012 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:06.958569 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:06.958582 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:06.958587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:06.958681 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:06.959114 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:06.959118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:06.959119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:06.959317 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:06.959395 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:06.960053 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 10:50:06.960208 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:50:06.961232 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:06.947948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.129:49661)
0927 10:50:06.948885 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:06.948945 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:06.950246 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:06.950778 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:06.950785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:06.951197 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:06.951681 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:06.951695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:06.952531 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:06.952534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:06.953393 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:06.953401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:06.953900 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:06.953904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:06.954073 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:06.954686 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:06.954702 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:06.956170 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:06.958120 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:06.958125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:06.958138 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:06.958451 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:06.958798 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:06.958803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:06.958805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:06.958994 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:06.959421 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:06.959431 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:06.959763 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 10:50:06.959978 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 10:50:06.960342 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":541,"threads_started":1}
I20260927 10:50:06.962494 12565 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36259
I20260927 10:50:06.962750 12565 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:06.963286 12565 heartbeater.cc:511] Master 127.6.31.190:36259 requested a full tablet report, sending...
I20260927 10:50:06.964740 12374 ts_manager.cc:194] Registered new tserver with Master: 5acccd0f20d448e7b4414693d2073ca9 (127.6.31.129:36455)
I20260927 10:50:06.965858 12374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:49661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab.
Entry for principal oryx/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab.
I20260927 10:50:06.976141 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36259
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab
--principal=oryx/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:07.086831 12570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:07.087191 12570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:07.087285 12570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:07.091255 12570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:07.091308 12570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:07.091389 12570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:07.096534 12570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab
--principal=oryx/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:07.097491 12570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:07.098436 12570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:07.107739 12576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:07.107731 12578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:07.107734 12575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:07.107973 12570 server_base.cc:1034] running on GCE node
I20260927 10:50:07.108428 12570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:07.108997 12570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:07.110172 12570 hybrid_clock.cc:648] HybridClock initialized: now 1790506207110158 us; error 24 us; skew 500 ppm
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:07.112797 12570 init.cc:377] Logged in from keytab as oryx/127.6.31.130@KRBTEST.COM (short username oryx)
I20260927 10:50:07.113623 12570 webserver.cc:460] Webserver started at http://127.6.31.130:35947/ using document root <none> and password file <none>
I20260927 10:50:07.114122 12570 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:07.114177 12570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:07.114333 12570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:07.116305 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "5f4e8905592544d89e0069f132cf39d5"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "3e6655695840d00d527abcbc10de2e6d"
server_key_iv: "628490ccf2da370f9e5662b23986a6d4"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.116811 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "5f4e8905592544d89e0069f132cf39d5"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "3e6655695840d00d527abcbc10de2e6d"
server_key_iv: "628490ccf2da370f9e5662b23986a6d4"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.120362 12570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:07.122659 12585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.123960 12570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:50:07.124053 12570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "5f4e8905592544d89e0069f132cf39d5"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "3e6655695840d00d527abcbc10de2e6d"
server_key_iv: "628490ccf2da370f9e5662b23986a6d4"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.124183 12570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:07.134981 12570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:07.137758 12570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:07.137943 12570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:07.138484 12570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:07.139290 12570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:07.139338 12570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.139400 12570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:07.139433 12570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.148044 12570 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:38551
I20260927 10:50:07.148080 12698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:38551 every 8 connection(s)
I20260927 10:50:07.148931 12570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:07.152575 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 12570
I20260927 10:50:07.152697 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:50:07.152999 6270 external_mini_cluster.cc:1428] Setting key 144c7f43726afa27785096963af40447
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:50:07.163673 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.150831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:36407 (local address 127.6.31.190:36259)
0927 10:50:07.150980 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.150984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.151926 (+ 942us) server_negotiation.cc:407] Connection header received
0927 10:50:07.153456 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.153460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.153519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.153627 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.154759 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.155367 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.157055 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.157275 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.157385 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.159959 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.159963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.159965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.159988 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.161488 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.162043 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.162050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.162053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.162150 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.162511 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.162515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.162517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.162698 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.162797 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.163383 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.163500 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:50:07.164534 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.151235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:36407)
0927 10:50:07.151770 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.151812 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.153219 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.153799 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.153806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.154156 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.154620 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.154633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.155497 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.155500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.156930 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.156944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.157420 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.157424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.157600 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.158207 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.158246 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.159791 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.161629 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.161635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.161651 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.161924 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.162267 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.162271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.162274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.162415 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.162805 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.162814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.163061 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.163331 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.163652 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.131@KRBTEST.COM" created.
I20260927 10:50:07.165745 12699 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36259
I20260927 10:50:07.166016 12699 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:07.166605 12699 heartbeater.cc:511] Master 127.6.31.190:36259 requested a full tablet report, sending...
I20260927 10:50:07.167747 12374 ts_manager.cc:194] Registered new tserver with Master: 5f4e8905592544d89e0069f132cf39d5 (127.6.31.130:38551)
I20260927 10:50:07.168462 12374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:36407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab.
Entry for principal oryx/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab.
I20260927 10:50:07.183568 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36259
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab
--principal=oryx/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:07.291975 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:07.292253 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:07.292296 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:07.296458 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:07.296511 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:07.296604 12704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:07.301888 12704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab
--principal=oryx/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:07.302943 12704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:07.304049 12704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:07.314234 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:07.314205 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:07.314307 12704 server_base.cc:1034] running on GCE node
W20260927 10:50:07.314200 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:07.314947 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:07.315680 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:07.316885 12704 hybrid_clock.cc:648] HybridClock initialized: now 1790506207316847 us; error 49 us; skew 500 ppm
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:07.320125 12704 init.cc:377] Logged in from keytab as oryx/127.6.31.131@KRBTEST.COM (short username oryx)
I20260927 10:50:07.321127 12704 webserver.cc:460] Webserver started at http://127.6.31.131:39889/ using document root <none> and password file <none>
I20260927 10:50:07.321782 12704 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:07.321883 12704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:07.322127 12704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:07.324383 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "acf2afc70aceb688b950f99283bc46c2"
server_key_iv: "b6821a2205f2a9288ae8f4e2074ce271"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.325006 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "acf2afc70aceb688b950f99283bc46c2"
server_key_iv: "b6821a2205f2a9288ae8f4e2074ce271"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.329248 12704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:50:07.331810 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.333148 12704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:07.333317 12704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "acf2afc70aceb688b950f99283bc46c2"
server_key_iv: "b6821a2205f2a9288ae8f4e2074ce271"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.333437 12704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:07.346282 12704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:07.349164 12704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:07.349401 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:07.349992 12704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:07.350814 12704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:07.350863 12704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.350903 12704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:07.350953 12704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.359582 12704 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:38823
I20260927 10:50:07.359608 12832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:38823 every 8 connection(s)
I20260927 10:50:07.360445 12704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:07.369761 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 12704
I20260927 10:50:07.369892 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:50:07.370244 6270 external_mini_cluster.cc:1428] Setting key 86d885ed20e49ca2937ad3b8a9966ce8
I20260927 10:50:07.373842 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.362226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:46481 (local address 127.6.31.190:36259)
0927 10:50:07.362385 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.362390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.363304 (+ 914us) server_negotiation.cc:407] Connection header received
0927 10:50:07.364506 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.364509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.364556 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.364630 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.365725 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.366253 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.367413 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.367619 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.367694 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.370538 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.370541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.370544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.370566 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.372133 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.372617 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.372619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.372621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.372691 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.372978 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.372980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.372981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.373143 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.373231 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.373637 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.373726 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:50:07.374689 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.362633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.131:46481)
0927 10:50:07.363122 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.363182 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.364326 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.364797 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.364803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.365145 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.365601 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.365613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.366349 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.366351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.367280 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.367289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.367738 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.367741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.367876 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.368510 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.368525 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.370372 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.372282 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.372286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.372297 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.372536 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.372781 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.372783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.372784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.372903 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.373229 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.373234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.373444 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.373593 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.373841 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260927 10:50:07.375964 12833 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36259
I20260927 10:50:07.376222 12833 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:07.376711 12833 heartbeater.cc:511] Master 127.6.31.190:36259 requested a full tablet report, sending...
I20260927 10:50:07.377750 12374 ts_manager.cc:194] Registered new tserver with Master: 4c2bdfdc5bd449a1a91d7b32fe3ab147 (127.6.31.131:38823)
I20260927 10:50:07.378314 12374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:46481
I20260927 10:50:07.383846 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.31.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.189.keytab.
Entry for principal oryx/127.6.31.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.189.keytab.
I20260927 10:50:07.410374 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.189:33823
--webserver_interface=127.6.31.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.189.keytab
--principal=oryx/127.6.31.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:36259,127.6.31.189:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:07.519640 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:07.519913 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:07.519954 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:07.524896 12838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:07.524982 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:07.525012 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:07.525043 12838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:07.525068 12838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:07.530910 12838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:36259,127.6.31.189:33823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.189.keytab
--principal=oryx/127.6.31.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.189:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.31.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
W20260927 10:50:07.531179 12838 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.31.190:36259,127.6.31.189:33823'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 10:50:07.532130 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:07.533178 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:07.541641 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:07.541635 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:07.541641 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:07.541831 12838 server_base.cc:1034] running on GCE node
I20260927 10:50:07.542454 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:07.543613 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:07.544791 12838 hybrid_clock.cc:648] HybridClock initialized: now 1790506207544781 us; error 30 us; skew 500 ppm
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:07.547617 12838 init.cc:377] Logged in from keytab as oryx/127.6.31.189@KRBTEST.COM (short username oryx)
I20260927 10:50:07.548493 12838 webserver.cc:460] Webserver started at http://127.6.31.189:42437/ using document root <none> and password file <none>
I20260927 10:50:07.549037 12838 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:07.549120 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:07.549357 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:07.551404 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data/instance:
uuid: "e248e0d2525d4001b8b10cc54d8f56a6"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "9206546363453c0a27ec85557090f0de"
server_key_iv: "d7e49b6138de9dce1cc8a8bdd2c5bea3"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.551956 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal/instance:
uuid: "e248e0d2525d4001b8b10cc54d8f56a6"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "9206546363453c0a27ec85557090f0de"
server_key_iv: "d7e49b6138de9dce1cc8a8bdd2c5bea3"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.555737 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 10:50:07.558049 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.559417 12838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:50:07.559531 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal
uuid: "e248e0d2525d4001b8b10cc54d8f56a6"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "9206546363453c0a27ec85557090f0de"
server_key_iv: "d7e49b6138de9dce1cc8a8bdd2c5bea3"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.559628 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:07.569331 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:07.572438 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:07.572628 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:07.579262 12838 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.189:33823
I20260927 10:50:07.579541 12905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.189:33823 every 8 connection(s)
I20260927 10:50:07.580404 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
I20260927 10:50:07.583038 12906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:07.586087 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 12838
I20260927 10:50:07.586818 12906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.189@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.31.189@KRBTEST.COM
I20260927 10:50:07.598666 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.588672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.6.31.190:36259)
0927 10:50:07.588821 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.588826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.589062 (+ 236us) server_negotiation.cc:407] Connection header received
0927 10:50:07.589368 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.589371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.589421 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.589516 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.590349 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.590848 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.591937 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.592155 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.592233 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.594838 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.594842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.594845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.594874 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.596461 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.597067 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.597069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.597071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.597141 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.597437 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.597440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.597441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.597708 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.597806 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.598357 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.598484 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 10:50:07.599771 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.587596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.6.31.189:33823)
0927 10:50:07.587980 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.587998 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.588036 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:50:07.588817 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.588839 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.589197 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.589409 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.590126 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.591131 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.591930 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.592340 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.592444 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.595303 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.595308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.595317 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.595336 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.596981 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.597544 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.597548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.597549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.597612 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.597901 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.597904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.597906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.598166 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.598353 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.598558 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.599227 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":541,"server-negotiator.queue_time_us":257,"spinlock_wait_cycles":4096,"thread_start_us":111,"threads_started":1}
I20260927 10:50:07.599777 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.588539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:39564)
0927 10:50:07.588922 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.588968 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.589175 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.589569 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.589574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.589940 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.590219 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.590231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.590965 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.590969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.591822 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.591830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.592279 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.592283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.592440 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.593212 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.593227 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.594689 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.596564 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.596570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.596580 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.596981 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.597221 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.597223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.597225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.597340 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.597805 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.597809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.598081 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.598302 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.598705 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":62,"spinlock_wait_cycles":5248,"thread_start_us":123,"threads_started":1}
I20260927 10:50:07.601024 12906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.189@KRBTEST.COM for oryx/127.6.31.189@KRBTEST.COM
I20260927 10:50:07.608850 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.601453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.0.0.1:44360)
0927 10:50:07.601609 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.601623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.601692 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.601985 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.601987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.602105 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.602199 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.602204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.602837 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.602839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.603469 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.603477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.604435 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.604437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.604506 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.604905 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.604916 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.606035 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.607625 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.607628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.607629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.607841 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.608084 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.608086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.608088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.608188 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.608495 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.608497 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.608564 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.608642 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.608745 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 10:50:07.608927 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.601524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.6.31.189:33823)
0927 10:50:07.601808 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.601812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.601826 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:50:07.601857 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.601859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.601900 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.601959 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.602286 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.602755 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.603567 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.604342 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.604422 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.606147 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.606150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.606152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.606171 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.607371 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.607930 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.607932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.607934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.607997 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.608271 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.608273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.608274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.608421 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.608491 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.608687 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.608806 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":218}
I20260927 10:50:07.611744 12906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: Bootstrap starting.
I20260927 10:50:07.613863 12906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:07.614643 12906 log.cc:824] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:07.616446 12906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: No bootstrap required, opened a new log
I20260927 10:50:07.619261 12906 raft_consensus.cc:348] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } }
I20260927 10:50:07.619460 12906 raft_consensus.cc:374] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:07.619529 12906 raft_consensus.cc:729] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e248e0d2525d4001b8b10cc54d8f56a6, State: Initialized, Role: FOLLOWER
I20260927 10:50:07.619997 12906 consensus_queue.cc:260] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } }
I20260927 10:50:07.620494 12912 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } } }
I20260927 10:50:07.620606 12912 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:50:07.621260 12906 sys_catalog.cc:564] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:50:07.625778 12923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:50:07.625864 12923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:50:07.627619 12838 master_runner.cc:428] Detected that this master e248e0d2525d4001b8b10cc54d8f56a6 is joining an existing cluster
I20260927 10:50:07.627674 12838 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.31.189:33823
I20260927 10:50:07.637784 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.629612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.6.31.190:36259)
0927 10:50:07.629750 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.629753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.630354 (+ 601us) server_negotiation.cc:407] Connection header received
0927 10:50:07.630471 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.630473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.630515 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.630582 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.631135 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.631761 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.632952 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.633155 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.633229 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.634407 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.634412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.634414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.634441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.636068 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.636472 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.636476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.636478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.636572 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.636892 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.636896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.636898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.637109 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.637256 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.637504 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.637622 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:50:07.637847 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.629537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:39572)
0927 10:50:07.630265 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.630280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.630383 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.630619 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.630622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.630857 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.630984 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.630991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.631867 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.631872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.632846 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.632856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.633264 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.633267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.633361 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.633829 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.633843 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.634282 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.636172 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.636178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.636181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.636378 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.636683 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.636685 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.636687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.636796 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.637245 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.637248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.637335 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.637444 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.637624 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":92,"threads_started":1}
I20260927 10:50:07.642175 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.629949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.0.0.1:44376)
0927 10:50:07.633911 (+ 3962us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.633923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.634032 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.634279 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.634281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.634525 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.634661 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.634672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.635639 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.635642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.636643 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.636654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.637125 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.637128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.637234 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.638027 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.638037 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.638322 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.640664 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.640670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.640673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.640904 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.641182 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.641188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.641190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.641275 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.641655 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.641658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.641745 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.641866 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.641992 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3908,"thread_start_us":87,"threads_started":1}
I20260927 10:50:07.642191 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.630019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.6.31.189:33823)
0927 10:50:07.630119 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.630123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.634002 (+ 3879us) server_negotiation.cc:407] Connection header received
0927 10:50:07.634132 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.634137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.634180 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.634249 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.634776 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.635520 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.636753 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.637028 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.637126 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.638451 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.638455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.638459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.638485 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.640375 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.641014 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.641018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.641020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.641095 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.641354 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.641357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.641359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.641556 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.641654 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.641928 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.642059 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":40}
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506206, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.190@KRBTEST.COM for oryx/127.6.31.189@KRBTEST.COM
I20260927 10:50:07.651546 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.643445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.6.31.189:33823)
0927 10:50:07.643521 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.643524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.643536 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:50:07.643588 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.643590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.643625 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.643686 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.644395 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.644844 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.646004 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.646194 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.646273 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.648452 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.648456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.648458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.648484 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.649822 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.650405 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.650408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.650409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.650476 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.650739 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.650741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.650742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.650892 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.650966 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.651356 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.651439 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 10:50:07.651649 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.642724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.0.0.1:44392)
0927 10:50:07.643133 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.643196 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.643309 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.643743 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.643748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.644173 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.644292 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.644299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.644936 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.644939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.645905 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.645915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.646292 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.646296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.646401 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.646860 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.646874 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.648327 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.650000 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.650010 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.650016 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.650317 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.650561 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.650564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.650566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.650655 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.650987 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.650994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.651152 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.651300 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.651479 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260927 10:50:07.653079 12374 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.31.189:33823: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6"
member_type: NON_VOTER
last_known_addr {
host: "127.6.31.189"
port: 33823
}
attrs {
promote: true
}
}
dest_uuid: "1331601e10ce4de1915f1b5b26522952"
cas_config_opid_index: -1
I20260927 10:50:07.653802 12374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } }
I20260927 10:50:07.655668 12933 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e248e0d2525d4001b8b10cc54d8f56a6 (127.6.31.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } }
I20260927 10:50:07.656418 12880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:07.658241 12880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Refusing update from remote peer 1331601e10ce4de1915f1b5b26522952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 10:50:07.658941 12934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.6.31.188@KRBTEST.COM; defaulting to no policy
I20260927 10:50:07.664397 12933 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.31.189:33823
I20260927 10:50:07.664808 12838 master.cc:537] Master@127.6.31.189:33823 shutting down...
I20260927 10:50:07.665854 12935 mvcc.cc:205] Tried to move back new op lower bound from 7333913422689685504 to 7333913422655217664. Current Snapshot: MvccSnapshot[applied={T|T < 7333913422689685504}]
I20260927 10:50:07.666320 12880 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e248e0d2525d4001b8b10cc54d8f56a6 (127.6.31.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } }
I20260927 10:50:07.666754 12934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } }
I20260927 10:50:07.666855 12934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:07.668275 12942 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } }
I20260927 10:50:07.668401 12942 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:50:07.668497 12942 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } }
I20260927 10:50:07.668569 12942 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:50:07.668737 12912 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1331601e10ce4de1915f1b5b26522952. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } }
I20260927 10:50:07.668797 12912 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: LEARNER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.188@KRBTEST.COM" created.
I20260927 10:50:07.673429 12838 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 LEARNER]: Raft consensus shutting down.
I20260927 10:50:07.673739 12838 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 LEARNER]: Raft consensus is shut down!
I20260927 10:50:07.673837 12838 tablet_replica.cc:330] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: stopping tablet replica
I20260927 10:50:07.679129 12838 master.cc:552] Master@127.6.31.189:33823 shutdown complete.
I20260927 10:50:07.679188 12838 master_runner.cc:305] Clearing existing system tablet
I20260927 10:50:07.679417 12838 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:07.679471 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:07.681654 12838 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:50:07.682606 12945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.682791 12838 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:07.682901 12838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:07.682933 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal
uuid: "e248e0d2525d4001b8b10cc54d8f56a6"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "9206546363453c0a27ec85557090f0de"
server_key_iv: "d7e49b6138de9dce1cc8a8bdd2c5bea3"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.684659 12838 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:07.687772 12838 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:50:07.687862 12838 log.cc:1196] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 10:50:07.688915 12838 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: Deleting consensus metadata
I20260927 10:50:07.689090 12838 master_runner.cc:315] Copying system tablet from 127.6.31.190:36259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.188.keytab.
Entry for principal oryx/127.6.31.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.188.keytab.
I20260927 10:50:07.690197 12838 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: tablet copy: Beginning tablet copy session from remote peer at address 127.6.31.190:36259
I20260927 10:50:07.690575 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.188:45305
--webserver_interface=127.6.31.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.188.keytab
--principal=oryx/127.6.31.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:36259,127.6.31.189:33823,127.6.31.188:45305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:50:07.702489 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.693837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:39580)
0927 10:50:07.694120 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.694135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.694292 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.694628 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.694631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.694838 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.694937 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.694944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.696110 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.696114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.696958 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.696968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.697650 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.697653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.697744 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.698312 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.698327 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.698733 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.700759 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.700762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.700764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.701000 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.701361 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.701364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.701366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.701454 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.701942 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.701944 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.702040 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.702151 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.702292 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":88,"threads_started":1}
I20260927 10:50:07.702611 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.693715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.6.31.190:36259)
0927 10:50:07.693867 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.693874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.694216 (+ 342us) server_negotiation.cc:407] Connection header received
0927 10:50:07.694395 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.694398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.694444 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.694494 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.695064 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.695959 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.697090 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.697430 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.697497 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.698879 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.698883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.698886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.698908 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.700628 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.701115 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.701122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.701126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.701235 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.701554 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.701560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.701564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.701831 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.701955 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.702281 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.702412 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:50:07.703068 12394 tablet_copy_service.cc:140] P 1331601e10ce4de1915f1b5b26522952: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e248e0d2525d4001b8b10cc54d8f56a6 ({username='oryx', principal='oryx/127.6.31.189@KRBTEST.COM'} at 127.0.0.1:39580)
I20260927 10:50:07.703317 12394 tablet_copy_service.cc:161] P 1331601e10ce4de1915f1b5b26522952: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e248e0d2525d4001b8b10cc54d8f56a6 at {username='oryx', principal='oryx/127.6.31.189@KRBTEST.COM'} at 127.0.0.1:39580: session id = e248e0d2525d4001b8b10cc54d8f56a6-00000000000000000000000000000000
I20260927 10:50:07.705092 12394 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:50:07.706670 12838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:50:07.708942 12838 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: tablet copy: Starting download of 0 data blocks...
I20260927 10:50:07.709131 12838 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: tablet copy: Starting download of 1 WAL segments...
I20260927 10:50:07.710759 12838 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:50:07.712587 12394 tablet_copy_service.cc:342] P 1331601e10ce4de1915f1b5b26522952: Request end of tablet copy session e248e0d2525d4001b8b10cc54d8f56a6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.31.189@KRBTEST.COM'} at 127.0.0.1:39580
I20260927 10:50:07.712698 12394 tablet_copy_service.cc:433] P 1331601e10ce4de1915f1b5b26522952: ending tablet copy session e248e0d2525d4001b8b10cc54d8f56a6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e248e0d2525d4001b8b10cc54d8f56a6
I20260927 10:50:07.719966 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:07.721832 12953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:07.724198 12954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:07.724279 12838 server_base.cc:1034] running on GCE node
W20260927 10:50:07.724380 12956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:07.724702 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:07.724982 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:07.726104 12838 hybrid_clock.cc:648] HybridClock initialized: now 1790506207726092 us; error 28 us; skew 500 ppm
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:07.727767 12838 init.cc:377] Logged in from keytab as oryx/127.6.31.189@KRBTEST.COM (short username oryx)
I20260927 10:50:07.728232 12838 webserver.cc:460] Webserver started at http://127.6.31.189:32815/ using document root <none> and password file <none>
I20260927 10:50:07.728454 12838 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:07.728497 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:07.729902 12838 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:07.730847 12963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.731070 12838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:07.731199 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal
uuid: "e248e0d2525d4001b8b10cc54d8f56a6"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "9206546363453c0a27ec85557090f0de"
server_key_iv: "d7e49b6138de9dce1cc8a8bdd2c5bea3"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.731271 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:07.758898 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:07.761170 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:07.766458 12838 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.189:33823
I20260927 10:50:07.766485 13015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.189:33823 every 8 connection(s)
I20260927 10:50:07.767246 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
I20260927 10:50:07.769168 13016 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:50:07.770224 13016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: Bootstrap starting.
I20260927 10:50:07.777318 13016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:50:07.777637 13016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: Bootstrap complete.
I20260927 10:50:07.778206 13016 raft_consensus.cc:348] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } }
I20260927 10:50:07.778333 13016 raft_consensus.cc:729] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e248e0d2525d4001b8b10cc54d8f56a6, State: Initialized, Role: LEARNER
I20260927 10:50:07.778431 13016 consensus_queue.cc:260] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } }
I20260927 10:50:07.778748 13019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } }
I20260927 10:50:07.778898 13016 sys_catalog.cc:564] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:07.778930 13019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:50:07.782258 13030 catalog_manager.cc:1259] Loaded cluster ID: 78e040796fe641b3a1d79c4740cdb9b0
I20260927 10:50:07.782359 13030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: loading cluster ID for follower catalog manager: success
I20260927 10:50:07.784528 13030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: acquiring CA information for follower catalog manager: success
I20260927 10:50:07.785899 13030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 10:50:07.815898 12951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:07.816190 12951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:07.816259 12951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:07.820271 12951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:07.820348 12951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:07.820382 12951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:07.820437 12951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:07.820470 12951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:07.825965 12951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:36259,127.6.31.189:33823,127.6.31.188:45305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.188.keytab
--principal=oryx/127.6.31.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.188:45305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.31.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:07.827060 12951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:07.828122 12951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:07.836357 13038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:07.836365 13035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:07.836472 12951 server_base.cc:1034] running on GCE node
W20260927 10:50:07.836395 13036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:07.837270 12951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:07.838481 12951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:07.839671 12951 hybrid_clock.cc:648] HybridClock initialized: now 1790506207839657 us; error 29 us; skew 500 ppm
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:07.842399 12951 init.cc:377] Logged in from keytab as oryx/127.6.31.188@KRBTEST.COM (short username oryx)
I20260927 10:50:07.843277 12951 webserver.cc:460] Webserver started at http://127.6.31.188:39217/ using document root <none> and password file <none>
I20260927 10:50:07.843775 12951 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:07.843828 12951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:07.843981 12951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:07.845862 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data/instance:
uuid: "e575508cb3934882849930d9b24ed899"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "d10e4049dbb254e12f6f89846275fadb"
server_key_iv: "a52f9be37e126fdb05d50ff11c882b0b"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.846403 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal/instance:
uuid: "e575508cb3934882849930d9b24ed899"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "d10e4049dbb254e12f6f89846275fadb"
server_key_iv: "a52f9be37e126fdb05d50ff11c882b0b"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.849875 12951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:50:07.852159 13045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:07.853359 12951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:50:07.853502 12951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal
uuid: "e575508cb3934882849930d9b24ed899"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "d10e4049dbb254e12f6f89846275fadb"
server_key_iv: "a52f9be37e126fdb05d50ff11c882b0b"
server_key_version: "encryptionkey@0"
I20260927 10:50:07.853608 12951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:07.867705 12951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:07.870930 12951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:07.871243 12951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:07.879060 12951 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.188:45305
I20260927 10:50:07.879379 13097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.188:45305 every 8 connection(s)
I20260927 10:50:07.880291 12951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
I20260927 10:50:07.883035 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:07.886611 13098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:50:07.887671 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 12951
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.188@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.31.188@KRBTEST.COM
I20260927 10:50:07.899695 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.888556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.6.31.190:36259)
0927 10:50:07.888684 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.888689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.889468 (+ 779us) server_negotiation.cc:407] Connection header received
0927 10:50:07.890305 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.890308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.890370 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.890456 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.891239 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.891770 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.892748 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.892955 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.893060 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.895658 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.895665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.895670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.895700 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.897439 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.897932 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.897939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.897943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.898032 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.898632 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.898638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.898640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.898824 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.898929 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.899403 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.899523 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:50:07.900540 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.889606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.6.31.188:45305)
0927 10:50:07.889842 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.889851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.889880 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:50:07.890104 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.890117 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.890290 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.890472 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.891314 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.892123 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.892929 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.893275 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.893379 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.896150 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.896170 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.896187 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.896212 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.898039 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.898391 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.898395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.898396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.898497 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.898776 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.898781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.898783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.899204 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.899399 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.899569 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.899702 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260927 10:50:07.900540 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.888893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:39590)
0927 10:50:07.889332 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.889372 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.890106 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.890504 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.890509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.890837 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.891062 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.891074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.891891 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.891894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.892626 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.892634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.893090 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.893093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.893251 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.893841 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.893865 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.895503 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.897556 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.897562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.897570 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.897838 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.898403 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.898406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.898407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.898532 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.898925 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.898931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.899186 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.899348 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.899643 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260927 10:50:07.902081 13098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.188@KRBTEST.COM for oryx/127.6.31.189@KRBTEST.COM
I20260927 10:50:07.910291 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.902611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.0.0.1:44398)
0927 10:50:07.903009 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.903021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.903120 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.903338 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.903340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.903474 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.903564 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.903568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.904518 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.904521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.905136 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.905144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.905588 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.905591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.905665 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.906084 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.906095 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.907267 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.908917 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.908920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.908922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.909158 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.909393 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.909395 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.909396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.909466 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.909885 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.909887 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.909968 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.910064 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.910164 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329}
I20260927 10:50:07.910333 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.902794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44398 (local address 127.6.31.189:33823)
0927 10:50:07.903052 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.903055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.903067 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:50:07.903199 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.903202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.903242 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.903315 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.903670 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.904418 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.905240 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.905459 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.905573 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.907399 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.907401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.907403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.907425 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.908815 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.909248 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.909251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.909252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.909321 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.909548 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.909550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.909551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.909718 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.909791 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.910108 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.910203 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
I20260927 10:50:07.911230 13098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506207, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.188@KRBTEST.COM for oryx/127.6.31.188@KRBTEST.COM
I20260927 10:50:07.918028 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.911659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.0.0.1:54172)
0927 10:50:07.911769 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.911783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.911854 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.912065 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.912067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.912186 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.912255 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.912259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.912898 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.912900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.913376 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.913382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.913765 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.913767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.913820 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.914184 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.914194 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.915314 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.916873 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.916876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.916877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.917040 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.917272 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.917274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.917275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.917338 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.917633 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.917635 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.917700 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.917786 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.917872 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 10:50:07.918040 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.911809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54172 (local address 127.6.31.188:45305)
0927 10:50:07.911908 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.911911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.911921 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:50:07.911947 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.911949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.911980 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.912041 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.912342 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.912811 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.913462 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.913682 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.913754 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.915425 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.915428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.915430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.915450 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.916790 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.917112 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.917114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.917115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.917201 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.917417 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.917419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.917420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.917564 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.917633 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.917834 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.917914 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:50:07.921384 13098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Bootstrap starting.
I20260927 10:50:07.923704 13098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:07.924417 13098 log.cc:824] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:07.926208 13098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: No bootstrap required, opened a new log
I20260927 10:50:07.929358 13098 raft_consensus.cc:348] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } }
I20260927 10:50:07.929548 13098 raft_consensus.cc:374] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:07.929628 13098 raft_consensus.cc:729] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e575508cb3934882849930d9b24ed899, State: Initialized, Role: FOLLOWER
I20260927 10:50:07.930033 13098 consensus_queue.cc:260] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } }
I20260927 10:50:07.930859 13104 sys_catalog.cc:455] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } } }
I20260927 10:50:07.931000 13104 sys_catalog.cc:458] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:50:07.931209 13098 sys_catalog.cc:564] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:50:07.935842 13115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:50:07.935931 13115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:50:07.939013 12951 master_runner.cc:428] Detected that this master e575508cb3934882849930d9b24ed899 is joining an existing cluster
I20260927 10:50:07.939073 12951 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.31.188:45305
I20260927 10:50:07.951689 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.941434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.6.31.189:33823)
0927 10:50:07.941555 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.941559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.942012 (+ 453us) server_negotiation.cc:407] Connection header received
0927 10:50:07.942175 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.942178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.942224 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.942273 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.943255 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.944047 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.945423 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.945788 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.945863 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.947550 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.947555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.947558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.947583 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.949814 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.950279 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.950282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.950284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.950365 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.950649 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.950655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.950659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.950918 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.951031 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.951397 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.951523 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:50:07.951723 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.941656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.0.0.1:44414)
0927 10:50:07.941926 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.941940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.942071 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.942434 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.942437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.942696 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.943126 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.943134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.944187 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.944190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.945290 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.945300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.946333 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.946337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.946429 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.946899 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.946914 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.947402 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.949939 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.949942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.949944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.950172 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.950468 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.950470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.950472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.950561 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.951021 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.951024 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.951146 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.951323 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.951484 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":280,"spinlock_wait_cycles":10624,"thread_start_us":115,"threads_started":1}
I20260927 10:50:07.952499 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.941357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:39594)
0927 10:50:07.941755 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.941773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.941918 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.942328 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.942331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.942701 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.942903 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.942908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.944012 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.944015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.945035 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.945044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.947976 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.947980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.948080 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.948521 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.948535 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.948929 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.950889 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.950892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.950894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.951163 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.951460 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.951464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.951466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.951542 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.951979 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.951983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.952092 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.952196 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.952321 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":4,"spinlock_wait_cycles":5376,"thread_start_us":100,"threads_started":1}
I20260927 10:50:07.952960 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.941427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.6.31.190:36259)
0927 10:50:07.941546 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.941551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.941843 (+ 292us) server_negotiation.cc:407] Connection header received
0927 10:50:07.942039 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.942043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.942100 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.942156 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.943035 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.943896 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.947589 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.947865 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.947980 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.949113 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.949116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.949119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.949143 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.950786 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.951263 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.951268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.951271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.951360 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.951640 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.951643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.951644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.951880 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.951972 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.952704 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.952816 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:50:07.954260 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 12436
I20260927 10:50:07.962486 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:36455
--local_ip_for_outbound_sockets=127.6.31.129
--tserver_master_addrs=127.6.31.190:36259
--webserver_port=42201
--webserver_interface=127.6.31.129
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab
--principal=oryx/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.31.190:36259,127.6.31.189:33823
--tserver_master_addrs=127.6.31.190:36259,127.6.31.189:33823,127.6.31.188:45305 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:50:07 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506206, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.190@KRBTEST.COM for oryx/127.6.31.188@KRBTEST.COM
I20260927 10:50:07.971315 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.942222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.0.0.1:54180)
0927 10:50:07.942634 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.942648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.942778 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.942998 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.943002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.943243 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.943347 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.943354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.944248 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.944251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.945134 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.945141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.945712 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.945715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.945793 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.946313 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.946328 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.967488 (+ 21160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.969700 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.969705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.969707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.969952 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.970233 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.970236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.970238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.970310 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.970734 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.970736 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.970808 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.970907 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.971055 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":51,"spinlock_wait_cycles":13824,"thread_start_us":77,"threads_started":1}
I20260927 10:50:07.971376 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.957494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.0.0.1:54196)
0927 10:50:07.957665 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.957681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.957757 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.958364 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.958370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.958582 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:07.958741 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.958751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.959818 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.959824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.960863 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.960876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.961469 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.961475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.961610 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.962145 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:07.962156 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:07.963422 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:07.969791 (+ 6369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.969795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.969797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.970049 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.970386 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:07.970390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:07.970392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:07.970474 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:07.970875 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:07.970881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:07.970975 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.971054 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.971225 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 10:50:07.972143 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.942043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.6.31.188:45305)
0927 10:50:07.942140 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.942145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.942715 (+ 570us) server_negotiation.cc:407] Connection header received
0927 10:50:07.942854 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.942857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.942892 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.942971 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.943457 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.944131 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.945241 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.945508 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.945576 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.967645 (+ 22069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.967650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.967653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.967682 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.969575 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.970053 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.970056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.970058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.970131 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.970399 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.970401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.970403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.970621 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.970701 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.971932 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.972018 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:50:07.972213 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.957749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.6.31.188:45305)
0927 10:50:07.958089 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.958096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.958109 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:50:07.958189 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.958192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.958240 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.958324 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:07.958857 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.959703 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.960988 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.961340 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.961442 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.963593 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:07.963596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:07.963598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:07.963620 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:07.969657 (+ 6037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.970160 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.970166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.970171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.970276 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:07.970579 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:07.970582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:07.970583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:07.970767 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:07.971669 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.971750 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.971860 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"spinlock_wait_cycles":6016,"thread_start_us":116,"threads_started":1}
I20260927 10:50:07.973302 12374 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.31.188:45305: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e575508cb3934882849930d9b24ed899"
member_type: NON_VOTER
last_known_addr {
host: "127.6.31.188"
port: 45305
}
attrs {
promote: true
}
}
dest_uuid: "1331601e10ce4de1915f1b5b26522952"
cas_config_opid_index: 5
I20260927 10:50:07.973976 12374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } }
I20260927 10:50:07.975072 12933 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e575508cb3934882849930d9b24ed899 (127.6.31.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.976109 13071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:07.976603 12933 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.31.188:45305
I20260927 10:50:07.976790 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.976876 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:07.977252 12951 master.cc:537] Master@127.6.31.188:45305 shutting down...
I20260927 10:50:07.977555 13071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Refusing update from remote peer 1331601e10ce4de1915f1b5b26522952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:50:07.978518 12934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:50:07.980327 12361 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 10:50:07.981263 12361 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 -> Peer e575508cb3934882849930d9b24ed899 (127.6.31.188:45305): Couldn't send request to peer e575508cb3934882849930d9b24ed899. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 10:50:07.983341 12951 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:07.983624 12951 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:07.983636 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.975289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.0.0.1:44428)
0927 10:50:07.977700 (+ 2411us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.977725 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:07.977819 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:07.978217 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:07.978220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:07.978238 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:07.978324 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.978330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.979446 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.979450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.980753 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:07.980765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.983203 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.983207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:07.983307 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.983353 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 10:50:07.983426 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:50:07.983505 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2321}
I20260927 10:50:07.983747 12951 tablet_replica.cc:330] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: stopping tablet replica
I20260927 10:50:07.984078 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.975524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.6.31.189:33823)
0927 10:50:07.975671 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:07.975676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:07.977785 (+ 2109us) server_negotiation.cc:407] Connection header received
0927 10:50:07.977915 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:07.977921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:07.977989 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:07.978089 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:07.978443 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.979299 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.982183 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:07.982837 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:07.982949 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:07.983140 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:07.983879 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 10:50:07.983963 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:50:07.984548 12990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 LEARNER]: Refusing update from remote peer 1331601e10ce4de1915f1b5b26522952: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:50:07.984998 12934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 10:50:07.986320 12990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e575508cb3934882849930d9b24ed899 (127.6.31.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.988126 13019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1331601e10ce4de1915f1b5b26522952. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.988234 13019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:50:07.988332 13019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: NON_VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: true } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.988401 13019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:50:07.989497 12934 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: attempting to promote NON_VOTER e248e0d2525d4001b8b10cc54d8f56a6 to VOTER
I20260927 10:50:07.989979 12934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } }
I20260927 10:50:07.991508 12990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 LEARNER]: Refusing update from remote peer 1331601e10ce4de1915f1b5b26522952: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 10:50:07.991619 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.990939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.0.0.1:54200)
0927 10:50:07.991459 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.991539 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441}
I20260927 10:50:07.991988 13129 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 10:50:07.992566 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:07.992248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.0.0.1:54206)
0927 10:50:07.992451 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:07.992503 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260927 10:50:07.992676 12361 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 -> Peer e575508cb3934882849930d9b24ed899 (127.6.31.188:45305): Couldn't send request to peer e575508cb3934882849930d9b24ed899. Status: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:50:07.993486 12933 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e248e0d2525d4001b8b10cc54d8f56a6 (127.6.31.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.994728 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.994834 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:07.994971 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.995064 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:07.995648 12990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e248e0d2525d4001b8b10cc54d8f56a6 (127.6.31.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.996845 13128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:07.996965 13128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:50:08.002373 12951 master.cc:552] Master@127.6.31.188:45305 shutdown complete.
I20260927 10:50:08.002452 12951 master_runner.cc:305] Clearing existing system tablet
I20260927 10:50:08.002679 12951 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:08.002748 12951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:08.003970 12951 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 10:50:08.004884 13131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.005097 12951 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:08.005234 12951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 10:50:08.005268 12951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal
uuid: "e575508cb3934882849930d9b24ed899"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "d10e4049dbb254e12f6f89846275fadb"
server_key_iv: "a52f9be37e126fdb05d50ff11c882b0b"
server_key_version: "encryptionkey@0"
I20260927 10:50:08.007000 12951 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:08.008991 12951 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:50:08.009094 12951 log.cc:1196] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 10:50:08.009347 12951 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Deleting consensus metadata
I20260927 10:50:08.009541 12951 master_runner.cc:315] Copying system tablet from 127.6.31.190:36259
I20260927 10:50:08.010670 12951 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: tablet copy: Beginning tablet copy session from remote peer at address 127.6.31.190:36259
I20260927 10:50:08.018936 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.011332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.6.31.190:36259)
0927 10:50:08.011468 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.011472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.011677 (+ 205us) server_negotiation.cc:407] Connection header received
0927 10:50:08.011816 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.011819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.011869 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.011950 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.012445 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.013063 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.013866 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.014095 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.014201 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.015598 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.015602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.015604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.015629 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.017313 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.017770 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.017774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.017776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.017857 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.018120 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.018125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.018126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.018303 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.018401 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.018658 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.018776 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:50:08.018955 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.011269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:39604)
0927 10:50:08.011586 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.011598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.011724 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.011966 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.011968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.012184 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.012303 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.012311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.013180 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.013182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.013754 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.013760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.014205 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.014209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.014297 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.015060 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.015071 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.015461 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.017446 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.017450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.017452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.017672 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.017944 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.017949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.017951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.018033 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.018400 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.018403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.018499 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.018607 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.018758 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":85,"threads_started":1}
I20260927 10:50:08.023046 12394 tablet_copy_service.cc:140] P 1331601e10ce4de1915f1b5b26522952: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e575508cb3934882849930d9b24ed899 ({username='oryx', principal='oryx/127.6.31.188@KRBTEST.COM'} at 127.0.0.1:39604)
I20260927 10:50:08.023255 12394 tablet_copy_service.cc:161] P 1331601e10ce4de1915f1b5b26522952: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e575508cb3934882849930d9b24ed899 at {username='oryx', principal='oryx/127.6.31.188@KRBTEST.COM'} at 127.0.0.1:39604: session id = e575508cb3934882849930d9b24ed899-00000000000000000000000000000000
I20260927 10:50:08.025195 12394 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:50:08.028198 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:50:08.030673 12951 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: tablet copy: Starting download of 0 data blocks...
I20260927 10:50:08.030887 12951 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: tablet copy: Starting download of 1 WAL segments...
I20260927 10:50:08.032881 12951 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:50:08.034422 12394 tablet_copy_service.cc:342] P 1331601e10ce4de1915f1b5b26522952: Request end of tablet copy session e575508cb3934882849930d9b24ed899-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.31.188@KRBTEST.COM'} at 127.0.0.1:39604
I20260927 10:50:08.034520 12394 tablet_copy_service.cc:433] P 1331601e10ce4de1915f1b5b26522952: ending tablet copy session e575508cb3934882849930d9b24ed899-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e575508cb3934882849930d9b24ed899
I20260927 10:50:08.041715 12951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:08.043460 13138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:08.044277 13139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:08.044571 13141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:08.048031 12951 server_base.cc:1034] running on GCE node
I20260927 10:50:08.048200 12951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:08.048517 12951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:08.049649 12951 hybrid_clock.cc:648] HybridClock initialized: now 1790506208049651 us; error 48 us; skew 500 ppm
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:08.051167 12951 init.cc:377] Logged in from keytab as oryx/127.6.31.188@KRBTEST.COM (short username oryx)
I20260927 10:50:08.051667 12951 webserver.cc:460] Webserver started at http://127.6.31.188:41915/ using document root <none> and password file <none>
I20260927 10:50:08.051858 12951 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:08.051909 12951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:08.053020 12951 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:08.053930 13148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.054147 12951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:08.054243 12951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal
uuid: "e575508cb3934882849930d9b24ed899"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "d10e4049dbb254e12f6f89846275fadb"
server_key_iv: "a52f9be37e126fdb05d50ff11c882b0b"
server_key_version: "encryptionkey@0"
I20260927 10:50:08.054347 12951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:08.062840 12951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:08.064893 12951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:08.069751 12951 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.188:45305
I20260927 10:50:08.070417 12951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
I20260927 10:50:08.070825 13200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.188:45305 every 8 connection(s)
I20260927 10:50:08.072077 13201 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:50:08.072894 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Bootstrap starting.
I20260927 10:50:08.082177 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:50:08.082489 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Bootstrap complete.
I20260927 10:50:08.083065 13201 raft_consensus.cc:348] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } }
I20260927 10:50:08.083245 13201 raft_consensus.cc:729] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e575508cb3934882849930d9b24ed899, State: Initialized, Role: LEARNER
I20260927 10:50:08.083369 13201 consensus_queue.cc:260] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } }
I20260927 10:50:08.083711 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:08.083879 13201 sys_catalog.cc:564] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:08.083892 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:50:08.087594 13215 catalog_manager.cc:1259] Loaded cluster ID: 78e040796fe641b3a1d79c4740cdb9b0
I20260927 10:50:08.087667 13215 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: loading cluster ID for follower catalog manager: success
I20260927 10:50:08.089179 13215 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: acquiring CA information for follower catalog manager: success
I20260927 10:50:08.090252 13215 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 10:50:08.126632 13125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:08.127004 13125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:08.127105 13125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:08.133772 13125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:08.133864 13125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:08.133953 13125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:08.142002 13125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab
--principal=oryx/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:36455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=42201
--tserver_master_addrs=127.6.31.190:36259,127.6.31.189:33823,127.6.31.188:45305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:08.143368 13125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:08.144586 13125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:08.155246 13221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:08.155228 13220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:08.155480 13125 server_base.cc:1034] running on GCE node
W20260927 10:50:08.155324 13223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:08.155999 13125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:08.156672 13125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:08.157804 13125 hybrid_clock.cc:648] HybridClock initialized: now 1790506208157784 us; error 33 us; skew 500 ppm
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:08.161206 13125 init.cc:377] Logged in from keytab as oryx/127.6.31.129@KRBTEST.COM (short username oryx)
I20260927 10:50:08.162190 13125 webserver.cc:460] Webserver started at http://127.6.31.129:42201/ using document root <none> and password file <none>
I20260927 10:50:08.162786 13125 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:08.162894 13125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:08.166911 13125 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 10:50:08.169076 13230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.170480 13125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:50:08.170954 12699 heartbeater.cc:503] Master 127.6.31.190:36259 was elected leader, sending a full tablet report...
I20260927 10:50:08.170643 13125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "5acccd0f20d448e7b4414693d2073ca9"
format_stamp: "Formatted at 2026-09-27 10:50:06 on dist-test-slave-c60x"
server_key: "2b9a7219303d440c245cd6592e89a138"
server_key_iv: "2e7fb659dcb289c026f072f9a3fe2e8a"
server_key_version: "encryptionkey@0"
I20260927 10:50:08.171391 13125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:08.192914 13125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:08.196185 13125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:08.196398 13125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:08.197002 13125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:08.197911 13125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:08.197984 13125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.198050 13125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:08.198096 13125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.207731 13125 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:36455
I20260927 10:50:08.207751 13343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:36455 every 8 connection(s)
I20260927 10:50:08.208735 13125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:08.213562 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 13125
I20260927 10:50:08.213757 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 12570
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.188@KRBTEST.COM
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.189@KRBTEST.COM
I20260927 10:50:08.220855 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:38551
--local_ip_for_outbound_sockets=127.6.31.130
--tserver_master_addrs=127.6.31.190:36259
--webserver_port=35947
--webserver_interface=127.6.31.130
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab
--principal=oryx/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.31.190:36259,127.6.31.189:33823
--tserver_master_addrs=127.6.31.190:36259,127.6.31.189:33823,127.6.31.188:45305 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:50:08.225594 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.212062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:57269 (local address 127.6.31.188:45305)
0927 10:50:08.212317 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.212322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.212337 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:50:08.213215 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.213220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.213272 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.213357 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.214296 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.215391 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.216917 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.217283 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.217375 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.221537 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.221544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.221550 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.221582 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.223259 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.223803 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.223807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.223809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.223891 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.224353 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.224359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.224362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.224619 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.224721 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.225274 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.225414 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
I20260927 10:50:08.225816 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.210500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:43115 (local address 127.6.31.189:33823)
0927 10:50:08.210682 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.210688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.211584 (+ 896us) server_negotiation.cc:407] Connection header received
0927 10:50:08.212692 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.212695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.212754 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.212834 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.214627 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.215487 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.216848 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.217071 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.217181 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.221273 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.221277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.221280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.221332 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.223587 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.224107 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.224114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.224118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.224205 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.224538 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.224544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.224549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.224791 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.224889 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.225483 (+ 594us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.225627 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:50:08.226521 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.211934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.6.31.129:57269)
0927 10:50:08.212186 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.212201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.212512 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.213490 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.213495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.213652 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.214139 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.214164 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.215543 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.215546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.216785 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.216796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.217401 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.217405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.217542 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.218269 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.218298 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.220776 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.223385 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.223391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.223408 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.223702 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.224022 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.224027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.224030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.224218 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.224719 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.224726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.224982 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.225181 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.225567 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":69,"spinlock_wait_cycles":4480,"thread_start_us":77,"threads_started":1}
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:08.227589 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.210902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.6.31.129:43115)
0927 10:50:08.211433 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.211483 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.212512 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.212974 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.212981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.213437 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.214471 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.214482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.215916 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.215922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.216728 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.216737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.217211 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.217215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.217423 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.218295 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.218308 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.221147 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.223703 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.223706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.223708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.223990 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.224319 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.224321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.224324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.224419 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.224935 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.224941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.225060 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.225182 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.227313 (+ 2131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":1832,"spinlock_wait_cycles":29440,"thread_start_us":185,"threads_started":1}
I20260927 10:50:08.228628 13344 heartbeater.cc:347] Connected to a master server at 127.6.31.188:45305
I20260927 10:50:08.228811 13346 heartbeater.cc:347] Connected to a master server at 127.6.31.189:33823
I20260927 10:50:08.228888 13344 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:08.228899 13346 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:08.229569 13346 heartbeater.cc:511] Master 127.6.31.189:33823 requested a full tablet report, sending...
I20260927 10:50:08.230386 13344 heartbeater.cc:511] Master 127.6.31.188:45305 requested a full tablet report, sending...
I20260927 10:50:08.231172 12980 ts_manager.cc:194] Registered new tserver with Master: 5acccd0f20d448e7b4414693d2073ca9 (127.6.31.129:36455)
I20260927 10:50:08.231429 13161 ts_manager.cc:194] Registered new tserver with Master: 5acccd0f20d448e7b4414693d2073ca9 (127.6.31.129:36455)
I20260927 10:50:08.232267 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.210536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:44027 (local address 127.6.31.190:36259)
0927 10:50:08.210663 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.210667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.211586 (+ 919us) server_negotiation.cc:407] Connection header received
0927 10:50:08.212948 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.212951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.212994 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.213078 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.219455 (+ 6377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.220306 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.221698 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.222102 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.222218 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.227948 (+ 5730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.227952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.227954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.227986 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.229687 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.230178 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.230182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.230185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.230299 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.230905 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.230910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.230914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.231228 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.231332 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.231890 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.232036 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:50:08.232913 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.210902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.129:44027)
0927 10:50:08.211432 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.211484 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.212500 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.213096 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.213099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.213440 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.219270 (+ 5830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.219278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.220473 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.220477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.221352 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.221365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.222772 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.222777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.223773 (+ 996us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.226294 (+ 2521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.226305 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.227804 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.229813 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.229816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.229818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.230043 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.230430 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.230436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.230510 (+ 74us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.230614 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.231484 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.231488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.231667 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.232478 (+ 811us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.232655 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":74,"spinlock_wait_cycles":52352,"thread_start_us":250,"threads_started":1}
I20260927 10:50:08.233682 13345 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36259
I20260927 10:50:08.233777 13345 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:08.233942 13345 heartbeater.cc:511] Master 127.6.31.190:36259 requested a full tablet report, sending...
I20260927 10:50:08.234809 12374 ts_manager.cc:194] Re-registered known tserver with Master: 5acccd0f20d448e7b4414693d2073ca9 (127.6.31.129:36455)
I20260927 10:50:08.235666 12374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:44027
W20260927 10:50:08.342008 13352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:08.342272 13352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:08.342316 13352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:08.346313 13352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:08.346369 13352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:08.346426 13352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:08.351812 13352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab
--principal=oryx/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:38551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=35947
--tserver_master_addrs=127.6.31.190:36259,127.6.31.189:33823,127.6.31.188:45305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:08.352814 13352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:08.353753 13352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:50:08.364463 13352 server_base.cc:1034] running on GCE node
W20260927 10:50:08.364893 13358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:08.364881 13360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:08.364881 13357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:08.365451 13352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:08.366158 13352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:08.367323 13352 hybrid_clock.cc:648] HybridClock initialized: now 1790506208367308 us; error 28 us; skew 500 ppm
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:08.370276 13352 init.cc:377] Logged in from keytab as oryx/127.6.31.130@KRBTEST.COM (short username oryx)
I20260927 10:50:08.371207 13352 webserver.cc:460] Webserver started at http://127.6.31.130:35947/ using document root <none> and password file <none>
I20260927 10:50:08.371783 13352 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:08.371884 13352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:08.375946 13352 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:08.377997 13367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.379352 13352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:08.379482 13352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "5f4e8905592544d89e0069f132cf39d5"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "3e6655695840d00d527abcbc10de2e6d"
server_key_iv: "628490ccf2da370f9e5662b23986a6d4"
server_key_version: "encryptionkey@0"
I20260927 10:50:08.380025 13352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:08.380316 12833 heartbeater.cc:503] Master 127.6.31.190:36259 was elected leader, sending a full tablet report...
I20260927 10:50:08.408409 13352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:08.411576 13352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:08.411746 13352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:08.412318 13352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:08.413259 13352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:08.413308 13352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.413352 13352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:08.413369 13352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.423203 13352 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:38551
I20260927 10:50:08.423231 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:38551 every 8 connection(s)
I20260927 10:50:08.424543 13352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:08.433197 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 13352
I20260927 10:50:08.433364 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 12704
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.188@KRBTEST.COM
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.189@KRBTEST.COM
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:08.440729 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.428119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:43887 (local address 127.6.31.188:45305)
0927 10:50:08.428247 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.428251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.428349 (+ 98us) server_negotiation.cc:407] Connection header received
0927 10:50:08.428794 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.428797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.428854 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.428928 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.430194 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.430731 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.432559 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.432918 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.433013 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.436363 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.436367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.436370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.436399 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.438231 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.438851 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.438857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.438860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.438951 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.439407 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.439413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.439417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.439654 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.439724 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.440416 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.440558 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:50:08.441202 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.426450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:40711 (local address 127.6.31.190:36259)
0927 10:50:08.426577 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.426581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.427636 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 10:50:08.428820 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.428823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.428877 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.428931 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.430210 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.430996 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.433120 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.433458 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.433564 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.437762 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.437767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.437771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.437800 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.439461 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.439920 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.439923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.439925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.440007 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.440289 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.440295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.440299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.440503 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.440583 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.440930 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.441059 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:50:08.441636 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:38823
--local_ip_for_outbound_sockets=127.6.31.131
--tserver_master_addrs=127.6.31.190:36259
--webserver_port=39889
--webserver_interface=127.6.31.131
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab
--principal=oryx/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.31.190:36259,127.6.31.189:33823
--tserver_master_addrs=127.6.31.190:36259,127.6.31.189:33823,127.6.31.188:45305 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:50:08.442966 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.426930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:40711)
0927 10:50:08.427487 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.427538 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.428594 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.429125 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.429133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.429576 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.430058 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.430067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.431243 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.431246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.432455 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.432466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.433706 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.433712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.433830 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.434365 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.434381 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.437579 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.439573 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.439575 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.439578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.439816 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.440108 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.440111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.440113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.440187 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.440638 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.440641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.440734 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.441359 (+ 625us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.441853 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":324,"spinlock_wait_cycles":106368,"thread_start_us":154,"threads_started":1}
I20260927 10:50:08.443293 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.428011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.6.31.130:43887)
0927 10:50:08.428271 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.428282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.428606 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.429125 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.429132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.429512 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.430053 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.430065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.430911 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.430916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.432455 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.432466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.433015 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.433019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.433186 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.433971 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.434000 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.436052 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.438372 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.438382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.438400 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.438731 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.439068 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.439073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.439077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.439287 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.439808 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.439814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.440139 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.441359 (+ 1220us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.441898 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":360,"spinlock_wait_cycles":6400,"thread_start_us":88,"threads_started":1}
I20260927 10:50:08.443513 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.426374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:51565 (local address 127.6.31.189:33823)
0927 10:50:08.426515 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.426520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.427766 (+ 1246us) server_negotiation.cc:407] Connection header received
0927 10:50:08.428824 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.428827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.428877 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.428931 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.430210 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.431007 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.432390 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.432757 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.432851 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.437300 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.437307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.437313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.437347 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.439693 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.440196 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.440202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.440206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.440291 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.440640 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.440646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.440651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.440908 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.440992 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.443188 (+ 2196us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.443341 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:50:08.444119 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.426930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.6.31.130:51565)
0927 10:50:08.427676 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.427688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.428594 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.429126 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.429133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.429576 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.430058 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.430067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.431107 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.431110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.432280 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.432289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.432921 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.432926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.433190 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.434054 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.434072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.437149 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.439820 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.439827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.439832 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.440084 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.440417 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.440423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.440427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.440520 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.441052 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.441057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.442908 (+ 1851us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.443725 (+ 817us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.443886 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":294,"spinlock_wait_cycles":7040,"thread_start_us":164,"threads_started":1}
I20260927 10:50:08.444298 13482 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36259
I20260927 10:50:08.444360 13481 heartbeater.cc:347] Connected to a master server at 127.6.31.188:45305
I20260927 10:50:08.444540 13481 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:08.444540 13482 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:08.445103 13482 heartbeater.cc:511] Master 127.6.31.190:36259 requested a full tablet report, sending...
I20260927 10:50:08.445109 13481 heartbeater.cc:511] Master 127.6.31.188:45305 requested a full tablet report, sending...
I20260927 10:50:08.446095 13483 heartbeater.cc:347] Connected to a master server at 127.6.31.189:33823
I20260927 10:50:08.446183 13483 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:08.446305 12374 ts_manager.cc:194] Re-registered known tserver with Master: 5f4e8905592544d89e0069f132cf39d5 (127.6.31.130:38551)
I20260927 10:50:08.446372 13483 heartbeater.cc:511] Master 127.6.31.189:33823 requested a full tablet report, sending...
I20260927 10:50:08.447021 12374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:40711
I20260927 10:50:08.447039 13161 ts_manager.cc:194] Registered new tserver with Master: 5f4e8905592544d89e0069f132cf39d5 (127.6.31.130:38551)
I20260927 10:50:08.447070 12980 ts_manager.cc:194] Registered new tserver with Master: 5f4e8905592544d89e0069f132cf39d5 (127.6.31.130:38551)
I20260927 10:50:08.472116 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.468528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.6.31.188:45305)
0927 10:50:08.468672 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.468677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.468697 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:50:08.468798 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.468801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.468861 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.468962 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:08.469251 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.470016 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.470991 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.471630 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.471741 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.471889 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.471925 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.471987 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:50:08.472136 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.468382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.0.0.1:54216)
0927 10:50:08.468577 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.468597 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.468706 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.469001 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.469004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.469024 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:08.469110 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.469115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.470135 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.470138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.470869 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.470877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.471762 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.471765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.471832 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.471853 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.471906 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.471967 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 10:50:08.472843 13175 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 10:50:08.473393 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1331601e10ce4de1915f1b5b26522952. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:08.473546 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: This master's current role is: LEARNER
W20260927 10:50:08.556712 13488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:08.556960 13488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:08.557003 13488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:08.561180 13488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:08.561241 13488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:08.561299 13488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:08.566468 13488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab
--principal=oryx/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:38823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=39889
--tserver_master_addrs=127.6.31.190:36259,127.6.31.189:33823,127.6.31.188:45305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:08.567561 13488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:08.568585 13488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:08.577957 13496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:08.577965 13493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:08.577965 13494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:08.578270 13488 server_base.cc:1034] running on GCE node
I20260927 10:50:08.578809 13488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:08.579495 13488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:08.580655 13488 hybrid_clock.cc:648] HybridClock initialized: now 1790506208580621 us; error 46 us; skew 500 ppm
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:08.583451 13488 init.cc:377] Logged in from keytab as oryx/127.6.31.131@KRBTEST.COM (short username oryx)
I20260927 10:50:08.584410 13488 webserver.cc:460] Webserver started at http://127.6.31.131:39889/ using document root <none> and password file <none>
I20260927 10:50:08.585083 13488 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:08.585188 13488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:08.588971 13488 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260927 10:50:08.590982 13503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.592216 13488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 10:50:08.592343 13488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147"
format_stamp: "Formatted at 2026-09-27 10:50:07 on dist-test-slave-c60x"
server_key: "acf2afc70aceb688b950f99283bc46c2"
server_key_iv: "b6821a2205f2a9288ae8f4e2074ce271"
server_key_version: "encryptionkey@0"
I20260927 10:50:08.592873 13488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:08.603616 13488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:08.606894 13488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:08.607124 13488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:08.607710 13488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:08.608558 13488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:08.608623 13488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.608685 13488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:08.608731 13488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:08.617017 13488 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:38823
I20260927 10:50:08.617076 13616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:38823 every 8 connection(s)
I20260927 10:50:08.618028 13488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:50:08.619024 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 13488
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:08.634474 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.621088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.6.31.189:33823)
0927 10:50:08.621691 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.621697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.622424 (+ 727us) server_negotiation.cc:407] Connection header received
0927 10:50:08.622474 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.622477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.622525 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.622606 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.624599 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.625366 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.626582 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.626954 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.627029 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.628895 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.628899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.628902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.628928 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.630665 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.632158 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.632161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.632163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.632253 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.633090 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.633095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.633098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.633333 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.633447 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.633924 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.634052 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":64,"threads_started":1}
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.188@KRBTEST.COM
I20260927 10:50:08.635644 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.621415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.6.31.190:36259)
0927 10:50:08.622826 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.622831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.622842 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:50:08.622881 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.622884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.622928 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.623560 (+ 632us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.624359 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.625209 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.627904 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.628321 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.629106 (+ 785us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.630452 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.630456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.630459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.630485 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.633079 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.633836 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.633840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.633842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.633931 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.634240 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.634243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.634245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.634474 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.635260 (+ 786us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.635336 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.635462 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1340,"thread_start_us":78,"threads_started":1}
I20260927 10:50:08.636116 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.622254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.6.31.188:45305)
0927 10:50:08.622605 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.622609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.622842 (+ 233us) server_negotiation.cc:407] Connection header received
0927 10:50:08.622889 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.622892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.622944 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.623032 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.624497 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.625249 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.627372 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.627672 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.627792 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.630452 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.630456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.630459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.630484 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.633232 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.634545 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.634551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.634553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.634634 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.634908 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.634913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.634915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.635157 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.635279 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.635842 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.635970 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":88,"threads_started":1}
Sep 27 10:50:08 dist-test-slave-c60x krb5kdc[12329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506208, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.189@KRBTEST.COM
I20260927 10:50:08.640148 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.622008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:48353 (local address 127.6.31.188:45305)
0927 10:50:08.622456 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.622460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.622476 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:50:08.623661 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.623664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.623712 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.623761 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.625437 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.626535 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.631593 (+ 5058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.631966 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.632085 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.635519 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.635522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.635525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.635547 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.637551 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.638226 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.638231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.638234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.638318 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.638805 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.638811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.638815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.639047 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.639167 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.639869 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.639980 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379}
I20260927 10:50:08.641352 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.620038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:47479 (local address 127.6.31.189:33823)
0927 10:50:08.620526 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.620530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.622116 (+ 1586us) server_negotiation.cc:407] Connection header received
0927 10:50:08.623411 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.623417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.623473 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.623529 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.626372 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.627163 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.631304 (+ 4141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.631660 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.631728 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.637631 (+ 5903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.637635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.637638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.637659 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.639618 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.640143 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.640146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.640148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.640242 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.640527 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.640530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.640532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.640725 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.640801 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.641106 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.641209 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398}
I20260927 10:50:08.641384 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.620189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:42031 (local address 127.6.31.190:36259)
0927 10:50:08.620323 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.620328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.622262 (+ 1934us) server_negotiation.cc:407] Connection header received
0927 10:50:08.623453 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.623456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.623504 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.623576 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.626626 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.627365 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.629774 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.630163 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.630248 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.637599 (+ 7351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.637603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.637606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.637629 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.639679 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.640205 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.640209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.640210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.640286 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.640564 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.640567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.640569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.640748 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.640812 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.641123 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.641246 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:50:08.641782 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.621620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.6.31.131:48353)
0927 10:50:08.622233 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.622245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.623539 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.623876 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.623878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.625197 (+ 1319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.625318 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.625327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.627505 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.627509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.631506 (+ 3997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.631514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.632431 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.632436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.632552 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.633931 (+ 1379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.633945 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.635379 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.637680 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.637690 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.637711 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.638115 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.638465 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.638471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.638476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.638679 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.639239 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.639246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.639582 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.640289 (+ 707us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.640656 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":2797,"spinlock_wait_cycles":195200,"thread_start_us":87,"threads_started":1}
I20260927 10:50:08.641943 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.620523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.131:42031)
0927 10:50:08.621973 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.622034 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.623346 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.623814 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.623823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.624308 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.624935 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.624952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.628101 (+ 3149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.628104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.628967 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.628978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.630436 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.630439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.631810 (+ 1371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.632389 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.632412 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.634332 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.639798 (+ 5466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.639801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.639803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.640088 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.640392 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.640395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.640397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.640473 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.640879 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.640881 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.640965 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.641543 (+ 578us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.641732 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":1227,"spinlock_wait_cycles":89984,"thread_start_us":162,"threads_started":1}
I20260927 10:50:08.642179 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.620523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.6.31.131:47479)
0927 10:50:08.621977 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.622033 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.623177 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.623813 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.623820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.625697 (+ 1877us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:08.626240 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.626247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.627317 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.627320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.631177 (+ 3857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.631188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.631862 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.631865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.631952 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.632617 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:08.632632 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:08.637412 (+ 4780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:08.639736 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.639741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.639746 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.640039 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.640350 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:08.640352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:08.640354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:08.640433 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:08.640853 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:08.640856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:08.640937 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.641516 (+ 579us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.641665 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"spinlock_wait_cycles":29568,"thread_start_us":132,"threads_started":1}
I20260927 10:50:08.642951 13619 heartbeater.cc:347] Connected to a master server at 127.6.31.189:33823
I20260927 10:50:08.643168 13617 heartbeater.cc:347] Connected to a master server at 127.6.31.188:45305
I20260927 10:50:08.643200 13619 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:08.643256 13617 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:08.643705 13617 heartbeater.cc:511] Master 127.6.31.188:45305 requested a full tablet report, sending...
I20260927 10:50:08.643707 13619 heartbeater.cc:511] Master 127.6.31.189:33823 requested a full tablet report, sending...
I20260927 10:50:08.643687 13618 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36259
I20260927 10:50:08.643936 13618 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:08.644071 13618 heartbeater.cc:511] Master 127.6.31.190:36259 requested a full tablet report, sending...
I20260927 10:50:08.639416 12374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:50:08.644786 12373 ts_manager.cc:194] Re-registered known tserver with Master: 4c2bdfdc5bd449a1a91d7b32fe3ab147 (127.6.31.131:38823)
I20260927 10:50:08.645016 12980 ts_manager.cc:194] Registered new tserver with Master: 4c2bdfdc5bd449a1a91d7b32fe3ab147 (127.6.31.131:38823)
I20260927 10:50:08.645016 13161 ts_manager.cc:194] Registered new tserver with Master: 4c2bdfdc5bd449a1a91d7b32fe3ab147 (127.6.31.131:38823)
W20260927 10:50:08.645406 12374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:08.645444 12373 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:42031
I20260927 10:50:08.647887 13635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: NON_VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: true } } }
I20260927 10:50:08.648026 13635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:08.651296 13129 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: attempting to promote NON_VOTER e575508cb3934882849930d9b24ed899 to VOTER
I20260927 10:50:08.651614 13129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } }
I20260927 10:50:08.653036 12990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Refusing update from remote peer 1331601e10ce4de1915f1b5b26522952: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:50:08.653975 13174 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 LEARNER]: Refusing update from remote peer 1331601e10ce4de1915f1b5b26522952: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:50:08.653963 13635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 10:50:08.654454 13635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 10:50:08.655933 13636 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e575508cb3934882849930d9b24ed899 (127.6.31.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:08.656488 12990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e575508cb3934882849930d9b24ed899 (127.6.31.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:08.657554 13635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:08.657675 13635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:08.657612 13636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:08.657737 13636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:08.657744 13129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:08.657866 13129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:08.658119 13174 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e575508cb3934882849930d9b24ed899 (127.6.31.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:08.660384 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:08.660501 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:50:08.660930 13645 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1331601e10ce4de1915f1b5b26522952" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:08.661075 13645 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:50:08.666266 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.659537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36455 (local address 127.0.0.1:59974)
0927 10:50:08.659849 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.659865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.659945 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.661955 (+ 2010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.661958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.661980 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:08.662085 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.662091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.663550 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.663554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.664804 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.664813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.665874 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.665878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.665956 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.665977 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.666066 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.666131 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":55}
I20260927 10:50:08.666357 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.659677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.6.31.129:36455)
0927 10:50:08.661432 (+ 1755us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.661443 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.661469 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:50:08.661552 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.661560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.661792 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.661970 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:08.662220 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.663437 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.664906 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.665753 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.665858 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.665996 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.666095 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.666189 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1649,"thread_start_us":145,"threads_started":1}
I20260927 10:50:08.666553 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.660599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.6.31.131:38823)
0927 10:50:08.661358 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.661373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.661945 (+ 572us) server_negotiation.cc:407] Connection header received
0927 10:50:08.662046 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.662051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.662254 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.662398 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:08.662568 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.663637 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.664976 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.665759 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.665856 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.665996 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.666303 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.666406 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":109,"threads_started":1}
I20260927 10:50:08.666847 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.660100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38823 (local address 127.0.0.1:59920)
0927 10:50:08.661843 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.661861 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.661949 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.662375 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.662378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.662394 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:08.662480 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.662486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.663744 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.663748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.664845 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.664855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.666065 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.666068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.666153 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.666174 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.666252 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.666701 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1680,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260927 10:50:08.667336 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.660961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.6.31.130:38551)
0927 10:50:08.662193 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.662204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.662293 (+ 89us) server_negotiation.cc:407] Connection header received
0927 10:50:08.662449 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.662454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.662623 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.662777 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:08.662940 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.663855 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.665227 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.665762 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.665879 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.666013 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.667072 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.667204 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":87,"threads_started":1}
I20260927 10:50:08.666872 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.660131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:38551 (local address 127.0.0.1:49244)
0927 10:50:08.662211 (+ 2080us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.662226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.662313 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.662756 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.662758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.662774 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:08.662852 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.662857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.663982 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.663985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.665115 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.665122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.666502 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.666506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.666598 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.666623 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.666698 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.666763 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2025,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260927 10:50:08.669303 13551 tablet_service.cc:1467] Processing CreateTablet for tablet 773aa71301624b23a6eddfbde95dba52 (DEFAULT_TABLE table=test-table [id=15d9abd5cbf44d28a566570821de5780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:08.669324 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 773aa71301624b23a6eddfbde95dba52 (DEFAULT_TABLE table=test-table [id=15d9abd5cbf44d28a566570821de5780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:08.670241 13551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773aa71301624b23a6eddfbde95dba52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:08.669943 13415 tablet_service.cc:1467] Processing CreateTablet for tablet 773aa71301624b23a6eddfbde95dba52 (DEFAULT_TABLE table=test-table [id=15d9abd5cbf44d28a566570821de5780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:08.670326 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773aa71301624b23a6eddfbde95dba52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:08.670830 13415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773aa71301624b23a6eddfbde95dba52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:08.677237 13651 tablet_bootstrap.cc:492] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Bootstrap starting.
I20260927 10:50:08.678314 13652 tablet_bootstrap.cc:492] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: Bootstrap starting.
I20260927 10:50:08.679368 13653 tablet_bootstrap.cc:492] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: Bootstrap starting.
I20260927 10:50:08.679944 13651 tablet_bootstrap.cc:654] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:08.680639 13652 tablet_bootstrap.cc:654] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:08.680900 13651 log.cc:824] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:08.681200 13653 tablet_bootstrap.cc:654] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:08.681427 13652 log.cc:824] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:08.681908 13653 log.cc:824] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:08.683305 13652 tablet_bootstrap.cc:492] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: No bootstrap required, opened a new log
I20260927 10:50:08.683492 13652 ts_tablet_manager.cc:1397] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 10:50:08.684736 13653 tablet_bootstrap.cc:492] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: No bootstrap required, opened a new log
I20260927 10:50:08.684929 13653 ts_tablet_manager.cc:1397] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 10:50:08.687264 13652 raft_consensus.cc:348] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } }
I20260927 10:50:08.687546 13652 raft_consensus.cc:374] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:08.687599 13652 raft_consensus.cc:729] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5acccd0f20d448e7b4414693d2073ca9, State: Initialized, Role: FOLLOWER
I20260927 10:50:08.688063 13652 consensus_queue.cc:260] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } }
I20260927 10:50:08.689076 13652 ts_tablet_manager.cc:1428] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 10:50:08.689291 13653 raft_consensus.cc:348] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } }
I20260927 10:50:08.689512 13653 raft_consensus.cc:374] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:08.689556 13653 raft_consensus.cc:729] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f4e8905592544d89e0069f132cf39d5, State: Initialized, Role: FOLLOWER
I20260927 10:50:08.689934 13653 consensus_queue.cc:260] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } }
I20260927 10:50:08.690330 13345 heartbeater.cc:503] Master 127.6.31.190:36259 was elected leader, sending a full tablet report...
I20260927 10:50:08.690806 13653 ts_tablet_manager.cc:1428] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:50:08.691169 13482 heartbeater.cc:503] Master 127.6.31.190:36259 was elected leader, sending a full tablet report...
I20260927 10:50:08.694881 13651 tablet_bootstrap.cc:492] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: No bootstrap required, opened a new log
I20260927 10:50:08.695118 13651 ts_tablet_manager.cc:1397] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260927 10:50:08.700163 13651 raft_consensus.cc:348] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } }
I20260927 10:50:08.700409 13651 raft_consensus.cc:374] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:08.700470 13651 raft_consensus.cc:729] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2bdfdc5bd449a1a91d7b32fe3ab147, State: Initialized, Role: FOLLOWER
I20260927 10:50:08.700876 13651 consensus_queue.cc:260] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } }
I20260927 10:50:08.701570 13618 heartbeater.cc:503] Master 127.6.31.190:36259 was elected leader, sending a full tablet report...
I20260927 10:50:08.701843 13651 ts_tablet_manager.cc:1428] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
W20260927 10:50:08.710183 13347 tablet.cc:2369] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:50:08.786083 13659 raft_consensus.cc:482] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:08.786314 13659 raft_consensus.cc:504] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } }
I20260927 10:50:08.787488 13659 leader_election.cc:288] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5acccd0f20d448e7b4414693d2073ca9 (127.6.31.129:36455), 5f4e8905592544d89e0069f132cf39d5 (127.6.31.130:38551)
I20260927 10:50:08.791124 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.787733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:38551 (local address 127.6.31.131:49373)
0927 10:50:08.787965 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.787979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.788076 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.788336 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.788339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.788353 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:08.788469 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.788474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.789165 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.789170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.789927 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.789935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.790565 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.790569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.790748 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.790775 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.790863 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.790941 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":42}
I20260927 10:50:08.791170 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.787834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:49373 (local address 127.6.31.130:38551)
0927 10:50:08.787977 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.787983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.788036 (+ 53us) server_negotiation.cc:407] Connection header received
0927 10:50:08.788190 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.788193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.788244 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.788324 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:08.788574 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.789055 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.790048 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.790433 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.790558 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.790610 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.790877 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.790998 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 10:50:08.791436 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.787942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:55885 (local address 127.6.31.129:36455)
0927 10:50:08.788067 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.788070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.788082 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:50:08.788125 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.788127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.788171 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.788257 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:08.788519 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.789019 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.790018 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.790396 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.790543 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.790607 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.791177 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.791263 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:50:08.791690 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.787733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36455 (local address 127.6.31.131:55885)
0927 10:50:08.787836 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:08.787852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:08.787940 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:08.788278 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:08.788281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:08.788296 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:08.788418 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.788423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.789123 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.789129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.789901 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:08.789909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.790678 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.790682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:08.790819 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.790847 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:50:08.790928 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:50:08.791014 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260927 10:50:08.791986 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773aa71301624b23a6eddfbde95dba52" candidate_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f4e8905592544d89e0069f132cf39d5" is_pre_election: true
I20260927 10:50:08.792241 13298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773aa71301624b23a6eddfbde95dba52" candidate_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5acccd0f20d448e7b4414693d2073ca9" is_pre_election: true
I20260927 10:50:08.792290 13435 raft_consensus.cc:2454] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c2bdfdc5bd449a1a91d7b32fe3ab147 in term 0.
I20260927 10:50:08.792483 13298 raft_consensus.cc:2454] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c2bdfdc5bd449a1a91d7b32fe3ab147 in term 0.
I20260927 10:50:08.792727 13507 leader_election.cc:302] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2bdfdc5bd449a1a91d7b32fe3ab147, 5f4e8905592544d89e0069f132cf39d5; no voters:
I20260927 10:50:08.793023 13659 raft_consensus.cc:2781] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:08.793114 13659 raft_consensus.cc:482] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:08.793213 13659 raft_consensus.cc:3037] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:08.794232 13659 raft_consensus.cc:504] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } }
I20260927 10:50:08.794570 13659 leader_election.cc:288] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [CANDIDATE]: Term 1 election: Requested vote from peers 5acccd0f20d448e7b4414693d2073ca9 (127.6.31.129:36455), 5f4e8905592544d89e0069f132cf39d5 (127.6.31.130:38551)
I20260927 10:50:08.794903 13298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773aa71301624b23a6eddfbde95dba52" candidate_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5acccd0f20d448e7b4414693d2073ca9"
I20260927 10:50:08.795018 13298 raft_consensus.cc:3037] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:08.795972 13298 raft_consensus.cc:2454] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c2bdfdc5bd449a1a91d7b32fe3ab147 in term 1.
I20260927 10:50:08.796234 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773aa71301624b23a6eddfbde95dba52" candidate_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f4e8905592544d89e0069f132cf39d5"
I20260927 10:50:08.796257 13506 leader_election.cc:302] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2bdfdc5bd449a1a91d7b32fe3ab147, 5acccd0f20d448e7b4414693d2073ca9; no voters:
I20260927 10:50:08.796360 13435 raft_consensus.cc:3037] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:08.796418 13659 raft_consensus.cc:2781] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:08.796653 13659 raft_consensus.cc:686] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2bdfdc5bd449a1a91d7b32fe3ab147, State: Running, Role: LEADER
I20260927 10:50:08.796996 13659 consensus_queue.cc:237] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } }
I20260927 10:50:08.797610 13435 raft_consensus.cc:2454] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c2bdfdc5bd449a1a91d7b32fe3ab147 in term 1.
I20260927 10:50:08.800464 12374 catalog_manager.cc:5495] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c2bdfdc5bd449a1a91d7b32fe3ab147 (127.6.31.131). New cstate: current_term: 1 leader_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } health_report { overall_health: HEALTHY } } }
I20260927 10:50:08.819977 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.816096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.6.31.131:38823)
0927 10:50:08.816222 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.816225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.816242 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:50:08.816344 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.816346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.816400 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.816494 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:08.816728 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.817308 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.818410 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.818641 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.818719 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.819007 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:08.819536 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:08.819638 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.819794 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.819861 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:50:08.826088 13298 raft_consensus.cc:1261] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [term 1 FOLLOWER]: Refusing update from remote peer 4c2bdfdc5bd449a1a91d7b32fe3ab147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:08.826136 13435 raft_consensus.cc:1261] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [term 1 FOLLOWER]: Refusing update from remote peer 4c2bdfdc5bd449a1a91d7b32fe3ab147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:08.826794 13659 consensus_queue.cc:1035] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:08.827008 13660 consensus_queue.cc:1035] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:08.846480 12373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:08.846642 12373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:08.850881 12373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1331601e10ce4de1915f1b5b26522952: Sending DeleteTablet for 3 replicas of tablet 773aa71301624b23a6eddfbde95dba52
I20260927 10:50:08.851835 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet 773aa71301624b23a6eddfbde95dba52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:08 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:59974
I20260927 10:50:08.852310 13551 tablet_service.cc:1514] Processing DeleteTablet for tablet 773aa71301624b23a6eddfbde95dba52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:08 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:59920
I20260927 10:50:08.852910 13415 tablet_service.cc:1514] Processing DeleteTablet for tablet 773aa71301624b23a6eddfbde95dba52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:08 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:49244
I20260927 10:50:08.853315 13672 tablet_replica.cc:330] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: stopping tablet replica
I20260927 10:50:08.853422 13673 tablet_replica.cc:330] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: stopping tablet replica
I20260927 10:50:08.853754 13672 raft_consensus.cc:2229] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:08.854267 13672 raft_consensus.cc:2258] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:08.854337 13674 tablet_replica.cc:330] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: stopping tablet replica
I20260927 10:50:08.853755 13673 raft_consensus.cc:2229] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:50:08.854612 13674 raft_consensus.cc:2229] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:08.854998 13674 raft_consensus.cc:2258] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:08.855044 13673 raft_consensus.cc:2258] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:08.855285 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 12342
I20260927 10:50:08.857131 13673 ts_tablet_manager.cc:1905] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:08.857131 13672 ts_tablet_manager.cc:1905] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:08.857546 13674 ts_tablet_manager.cc:1905] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:08.859860 13673 ts_tablet_manager.cc:1918] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:08.859860 13672 ts_tablet_manager.cc:1918] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:08.859963 13672 log.cc:1196] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/773aa71301624b23a6eddfbde95dba52
I20260927 10:50:08.860314 13672 ts_tablet_manager.cc:1939] T 773aa71301624b23a6eddfbde95dba52 P 5acccd0f20d448e7b4414693d2073ca9: Deleting consensus metadata
I20260927 10:50:08.860499 13673 log.cc:1196] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/773aa71301624b23a6eddfbde95dba52
I20260927 10:50:08.860862 13674 ts_tablet_manager.cc:1918] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:08.860965 13674 log.cc:1196] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/773aa71301624b23a6eddfbde95dba52
I20260927 10:50:08.861292 13674 ts_tablet_manager.cc:1939] T 773aa71301624b23a6eddfbde95dba52 P 5f4e8905592544d89e0069f132cf39d5: Deleting consensus metadata
I20260927 10:50:08.862438 13673 ts_tablet_manager.cc:1939] T 773aa71301624b23a6eddfbde95dba52 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Deleting consensus metadata
W20260927 10:50:08.863194 13371 connection.cc:535] server connection from 127.0.0.1:49244 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 10:50:08.863387 13504 connection.cc:441] server connection from 127.0.0.1:59920 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59920 (request call id 1) could send its response
W20260927 10:50:08.863656 13232 connection.cc:535] server connection from 127.0.0.1:59974 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:50:08.872596 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.853976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.6.31.189:33823)
0927 10:50:08.854113 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.854118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.860372 (+ 6254us) server_negotiation.cc:407] Connection header received
0927 10:50:08.861076 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.861080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.861139 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.861197 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.863458 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.864302 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.865528 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.865853 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.865966 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.868085 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.868087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.868089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.868107 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.869867 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.870340 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.870343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.870345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.870430 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.870707 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.870710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.870712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.870929 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.871048 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.872273 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.872402 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:50:08.877679 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.866803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.6.31.189:33823)
0927 10:50:08.866928 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.866931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.867550 (+ 619us) server_negotiation.cc:407] Connection header received
0927 10:50:08.867750 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.867757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.867812 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.867904 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.868592 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.869405 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.870629 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.871003 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.871117 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.873833 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.873837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.873840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.873863 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.875810 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.876350 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.876354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.876356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.876437 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.876738 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.876742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.876743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.876947 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.877043 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.877374 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.877502 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:50:08.880800 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.867431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.6.31.188:45305)
0927 10:50:08.868014 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.868018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.868559 (+ 541us) server_negotiation.cc:407] Connection header received
0927 10:50:08.868772 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.868776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.868840 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.868902 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.869853 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.870612 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.871811 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.872162 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.872267 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.876339 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.876344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.876347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.876381 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.878311 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.879376 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.879381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.879385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.879499 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.879854 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.879858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.879859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.880081 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.880231 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.880467 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.880613 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494}
I20260927 10:50:08.881574 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:08.854034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.6.31.188:45305)
0927 10:50:08.854200 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:08.854206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:08.872061 (+ 17855us) server_negotiation.cc:407] Connection header received
0927 10:50:08.872190 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:08.872194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:08.872244 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:08.872318 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:08.873165 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.873873 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.874957 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:08.875389 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:08.875499 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:08.877152 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:08.877158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:08.877164 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:08.877198 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:08.879671 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.880108 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.880111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.880113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.880196 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:08.880472 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:08.880476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:08.880477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:08.880681 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:08.880814 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:08.881370 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 10:50:08.881471 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 10:50:09.839660 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:09.839137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:56189)
0927 10:50:09.839440 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:09.839538 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
W20260927 10:50:09.840047 13482 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:36259 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:09.841548 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:09.841139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.129:56647)
0927 10:50:09.841364 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:09.841449 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260927 10:50:09.841632 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:09.841139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.131:37963)
0927 10:50:09.841426 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:09.841515 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
W20260927 10:50:09.841835 13345 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:36259 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
W20260927 10:50:09.842087 13618 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:36259 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:10.351630 13685 raft_consensus.cc:482] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1331601e10ce4de1915f1b5b26522952)
I20260927 10:50:10.351881 13686 raft_consensus.cc:482] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1331601e10ce4de1915f1b5b26522952)
I20260927 10:50:10.351863 13685 raft_consensus.cc:504] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } }
I20260927 10:50:10.352065 13686 raft_consensus.cc:504] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } }
I20260927 10:50:10.352974 13685 leader_election.cc:288] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259), e575508cb3934882849930d9b24ed899 (127.6.31.188:45305)
I20260927 10:50:10.353129 13686 leader_election.cc:288] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259), e248e0d2525d4001b8b10cc54d8f56a6 (127.6.31.189:33823)
I20260927 10:50:10.353623 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.353111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:35744)
0927 10:50:10.353435 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.353512 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":80,"threads_started":1}
W20260927 10:50:10.353801 12964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:10.354115 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.353101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:35742)
0927 10:50:10.353619 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.354021 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":65,"threads_started":1}
W20260927 10:50:10.354358 13149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:10.354749 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.354504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:35762)
0927 10:50:10.354639 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.354687 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 10:50:10.354874 12964 leader_election.cc:334] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259): Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:10.355660 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.355424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:35768)
0927 10:50:10.355549 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.355600 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 10:50:10.355794 13149 leader_election.cc:334] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259): Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:10.357599 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.353783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.6.31.188:45305)
0927 10:50:10.354083 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:10.354087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:10.354101 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:50:10.354146 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:10.354149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:10.354207 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:10.354593 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:10.354688 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:10.355413 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:10.356664 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:10.357048 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:10.357154 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:10.357219 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:10.357419 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 10:50:10.357485 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":61,"threads_started":1}
I20260927 10:50:10.357600 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.353491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45305 (local address 127.0.0.1:37726)
0927 10:50:10.353736 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.353750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:10.353855 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:10.354334 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:10.354337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:10.354354 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:10.354475 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:10.354482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:10.355565 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:10.355569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:10.356528 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:10.356537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:10.357165 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:10.357169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:10.357295 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:10.357314 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:50:10.357402 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:50:10.357473 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":22,"thread_start_us":58,"threads_started":1}
I20260927 10:50:10.357744 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.353198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.6.31.189:33823)
0927 10:50:10.353475 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:10.353479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:10.353851 (+ 372us) server_negotiation.cc:407] Connection header received
0927 10:50:10.354047 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:10.354051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:10.354095 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:10.354154 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:10.354627 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:10.355452 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:10.356778 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:10.357203 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:10.357320 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:10.357375 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:10.357536 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:50:10.357602 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":196,"thread_start_us":49,"threads_started":1}
I20260927 10:50:10.357766 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.353101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33823 (local address 127.0.0.1:49956)
0927 10:50:10.353771 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.353783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:10.353926 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:10.354320 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:10.354323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:10.354340 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:10.354477 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:10.354484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:10.355576 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:10.355579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:10.356631 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:10.356640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:10.357321 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:10.357325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:10.357414 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:10.357440 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:50:10.357522 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:50:10.357593 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":140,"thread_start_us":74,"threads_started":1}
I20260927 10:50:10.358152 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e575508cb3934882849930d9b24ed899" is_pre_election: true
I20260927 10:50:10.358153 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e575508cb3934882849930d9b24ed899" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" is_pre_election: true
I20260927 10:50:10.358362 13174 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e248e0d2525d4001b8b10cc54d8f56a6 in term 1.
I20260927 10:50:10.358362 12990 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e575508cb3934882849930d9b24ed899 in term 1.
I20260927 10:50:10.358738 12967 leader_election.cc:302] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e248e0d2525d4001b8b10cc54d8f56a6, e575508cb3934882849930d9b24ed899; no voters: 1331601e10ce4de1915f1b5b26522952
I20260927 10:50:10.358958 13685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:50:10.359014 13685 raft_consensus.cc:482] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1331601e10ce4de1915f1b5b26522952)
I20260927 10:50:10.359052 13685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:50:10.359220 13152 leader_election.cc:302] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e248e0d2525d4001b8b10cc54d8f56a6, e575508cb3934882849930d9b24ed899; no voters: 1331601e10ce4de1915f1b5b26522952
I20260927 10:50:10.359428 13686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:50:10.359508 13686 raft_consensus.cc:482] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1331601e10ce4de1915f1b5b26522952)
I20260927 10:50:10.359582 13686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:50:10.360221 13685 raft_consensus.cc:504] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } }
I20260927 10:50:10.360605 13685 leader_election.cc:288] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [CANDIDATE]: Term 2 election: Requested vote from peers 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259), e575508cb3934882849930d9b24ed899 (127.6.31.188:45305)
I20260927 10:50:10.360592 13686 raft_consensus.cc:504] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } }
I20260927 10:50:10.361001 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.360685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:35782)
0927 10:50:10.360845 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.360899 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 10:50:10.360998 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e575508cb3934882849930d9b24ed899"
I20260927 10:50:10.361168 13174 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e248e0d2525d4001b8b10cc54d8f56a6 in current term 2: Already voted for candidate e575508cb3934882849930d9b24ed899 in this term.
I20260927 10:50:10.361291 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e575508cb3934882849930d9b24ed899" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e248e0d2525d4001b8b10cc54d8f56a6"
I20260927 10:50:10.361411 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.361071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:35786)
0927 10:50:10.361299 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.361352 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 10:50:10.361440 12990 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e575508cb3934882849930d9b24ed899 in current term 2: Already voted for candidate e248e0d2525d4001b8b10cc54d8f56a6 in this term.
I20260927 10:50:10.361621 13686 leader_election.cc:288] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 2 election: Requested vote from peers 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259), e248e0d2525d4001b8b10cc54d8f56a6 (127.6.31.189:33823)
I20260927 10:50:10.361668 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.361410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:35794)
0927 10:50:10.361558 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.361608 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 10:50:10.361778 12964 leader_election.cc:334] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259): Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:10.361842 12964 leader_election.cc:302] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e248e0d2525d4001b8b10cc54d8f56a6; no voters: 1331601e10ce4de1915f1b5b26522952, e575508cb3934882849930d9b24ed899
I20260927 10:50:10.361982 13685 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 10:50:10.362151 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.361944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:35800)
0927 10:50:10.362033 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.362085 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 10:50:10.362269 13149 leader_election.cc:334] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259): Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:10.362329 13149 leader_election.cc:302] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e575508cb3934882849930d9b24ed899; no voters: 1331601e10ce4de1915f1b5b26522952, e248e0d2525d4001b8b10cc54d8f56a6
I20260927 10:50:10.362478 13686 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 10:50:10.840969 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.840564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:56561)
0927 10:50:10.840824 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.840880 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":105,"threads_started":1}
I20260927 10:50:10.842999 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.842556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.131:49379)
0927 10:50:10.842830 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.842902 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":63,"threads_started":1}
I20260927 10:50:10.843076 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:10.842352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.129:47511)
0927 10:50:10.842893 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:10.842960 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":80,"threads_started":1}
I20260927 10:50:11.842170 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:11.841605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:39021)
0927 10:50:11.841969 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:11.842039 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
I20260927 10:50:11.844018 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:11.843550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.131:45455)
0927 10:50:11.843849 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:11.843909 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":66,"threads_started":1}
I20260927 10:50:11.844086 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:11.843693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.129:43325)
0927 10:50:11.843915 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:11.843981 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260927 10:50:12.261083 13701 raft_consensus.cc:482] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:12.261226 13701 raft_consensus.cc:504] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } }
I20260927 10:50:12.261793 13701 leader_election.cc:288] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259), e248e0d2525d4001b8b10cc54d8f56a6 (127.6.31.189:33823)
I20260927 10:50:12.262282 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.261876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:36102)
0927 10:50:12.262139 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.262199 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260927 10:50:12.262305 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e575508cb3934882849930d9b24ed899" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" is_pre_election: true
I20260927 10:50:12.262465 12990 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e575508cb3934882849930d9b24ed899 in term 2.
I20260927 10:50:12.262800 13152 leader_election.cc:302] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e248e0d2525d4001b8b10cc54d8f56a6, e575508cb3934882849930d9b24ed899; no voters:
I20260927 10:50:12.262967 13701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 10:50:12.263026 13701 raft_consensus.cc:482] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:12.263072 13701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:50:12.263233 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.262900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:36110)
0927 10:50:12.263023 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.263079 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:50:12.263402 13149 leader_election.cc:334] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259): Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:12.264230 13701 raft_consensus.cc:504] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } }
I20260927 10:50:12.264576 13701 leader_election.cc:288] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 3 election: Requested vote from peers 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259), e248e0d2525d4001b8b10cc54d8f56a6 (127.6.31.189:33823)
I20260927 10:50:12.264905 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.264684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:36116)
0927 10:50:12.264795 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.264846 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 10:50:12.264943 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e575508cb3934882849930d9b24ed899" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e248e0d2525d4001b8b10cc54d8f56a6"
I20260927 10:50:12.265106 12990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:50:12.265538 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.265367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:36120)
0927 10:50:12.265445 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.265488 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 10:50:12.265627 13149 leader_election.cc:334] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259): Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
I20260927 10:50:12.266273 12990 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e575508cb3934882849930d9b24ed899 in term 3.
I20260927 10:50:12.266582 13152 leader_election.cc:302] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e248e0d2525d4001b8b10cc54d8f56a6, e575508cb3934882849930d9b24ed899; no voters: 1331601e10ce4de1915f1b5b26522952
I20260927 10:50:12.266746 13701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:50:12.266937 13701 raft_consensus.cc:686] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [term 3 LEADER]: Becoming Leader. State: Replica: e575508cb3934882849930d9b24ed899, State: Running, Role: LEADER
I20260927 10:50:12.267263 13701 consensus_queue.cc:237] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } }
I20260927 10:50:12.268419 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e575508cb3934882849930d9b24ed899. Latest consensus state: current_term: 3 leader_uuid: "e575508cb3934882849930d9b24ed899" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:12.268536 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:12.268851 13706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:12.270990 13706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:12.271426 13706 catalog_manager.cc:1259] Loaded cluster ID: 78e040796fe641b3a1d79c4740cdb9b0
I20260927 10:50:12.271471 13706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:12.272020 13706 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:12.272406 13706 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Loaded TSK: 0
I20260927 10:50:12.273067 13706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:50:12.341349 13161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:50:12.344590 13161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:12.347576 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.347274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:36128)
0927 10:50:12.347420 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.347482 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:50:12.347602 12990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [term 3 FOLLOWER]: Refusing update from remote peer e575508cb3934882849930d9b24ed899: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 10:50:12.348147 13704 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 10:50:12.348304 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.348073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.0.0.1:36130)
0927 10:50:12.348176 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.348228 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:50:12.348404 13149 consensus_peers.cc:487] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 -> Peer 1331601e10ce4de1915f1b5b26522952 (127.6.31.190:36259): Couldn't send request to peer 1331601e10ce4de1915f1b5b26522952. Status: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:50:12.349512 13708 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e575508cb3934882849930d9b24ed899. Latest consensus state: current_term: 3 leader_uuid: "e575508cb3934882849930d9b24ed899" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:12.349656 13708 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:50:12.350775 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e575508cb3934882849930d9b24ed899" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:12.350884 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:12.351191 13708 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e575508cb3934882849930d9b24ed899" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1331601e10ce4de1915f1b5b26522952" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36259 } } peers { permanent_uuid: "e248e0d2525d4001b8b10cc54d8f56a6" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33823 } attrs { promote: false } } peers { permanent_uuid: "e575508cb3934882849930d9b24ed899" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45305 } attrs { promote: false } } }
I20260927 10:50:12.351317 13708 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248e0d2525d4001b8b10cc54d8f56a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:50:12.361577 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.357060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36455 (local address 127.0.0.1:49278)
0927 10:50:12.357422 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.357434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:12.357514 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:12.358155 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:12.358158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:12.358174 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:12.358389 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.358394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.359367 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.359370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.360464 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.360470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.361110 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.361113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.361217 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.361244 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:50:12.361325 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:50:12.361400 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":97,"thread_start_us":92,"threads_started":1}
I20260927 10:50:12.361557 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.357389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.6.31.129:36455)
0927 10:50:12.357630 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:12.357633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:12.357645 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:50:12.357699 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:12.357701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:12.357738 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:12.358034 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:12.358507 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.359181 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.360566 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.360997 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.361113 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.361181 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:12.361341 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 10:50:12.361417 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":96,"threads_started":1}
I20260927 10:50:12.361604 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.356467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:38551 (local address 127.0.0.1:58982)
0927 10:50:12.356582 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.356598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:12.356694 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:12.357799 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:12.357802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:12.357822 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:12.357930 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.357936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.358887 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.358889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.360291 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.360300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.361197 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.361201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.361313 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.361344 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:50:12.361390 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 10:50:12.361459 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260927 10:50:12.362264 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.356863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.6.31.130:38551)
0927 10:50:12.357559 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:12.357564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:12.357581 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:50:12.357637 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:12.357639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:12.357683 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:12.357765 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:12.358043 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.358772 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.360419 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.361066 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.361928 (+ 862us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.362002 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:12.362039 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 10:50:12.362133 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":58,"threads_started":1}
I20260927 10:50:12.362525 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.356734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38823 (local address 127.0.0.1:45780)
0927 10:50:12.357290 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.357306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:12.357394 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:12.358171 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:12.358174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:12.358185 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:12.358388 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.358394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.360085 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.360088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.361247 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.361255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.362146 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.362150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.362254 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.362279 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:50:12.362332 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 10:50:12.362406 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":39,"thread_start_us":110,"threads_started":1}
I20260927 10:50:12.362516 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 7077b25ac79145edb3a1673b8ceac691 (DEFAULT_TABLE table=test-table [id=cf34393bdb7d45f5990ff248e2a118e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:12.362712 13415 tablet_service.cc:1467] Processing CreateTablet for tablet 7077b25ac79145edb3a1673b8ceac691 (DEFAULT_TABLE table=test-table [id=cf34393bdb7d45f5990ff248e2a118e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:12.362815 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7077b25ac79145edb3a1673b8ceac691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:12.362937 13415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7077b25ac79145edb3a1673b8ceac691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:12.363036 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.356842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.6.31.131:38823)
0927 10:50:12.357943 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:12.357946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:12.357957 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:50:12.358020 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:12.358023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:12.358071 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:12.358154 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:12.359309 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.359955 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.361389 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.362021 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.362695 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.362781 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:12.362817 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 10:50:12.362893 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":66,"threads_started":1}
I20260927 10:50:12.365810 13720 tablet_bootstrap.cc:492] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: Bootstrap starting.
I20260927 10:50:12.365926 13551 tablet_service.cc:1467] Processing CreateTablet for tablet 7077b25ac79145edb3a1673b8ceac691 (DEFAULT_TABLE table=test-table [id=cf34393bdb7d45f5990ff248e2a118e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:12.366166 13719 tablet_bootstrap.cc:492] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9: Bootstrap starting.
I20260927 10:50:12.366242 13551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7077b25ac79145edb3a1673b8ceac691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:12.366896 13720 tablet_bootstrap.cc:654] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:12.367559 13719 tablet_bootstrap.cc:654] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:12.368129 13720 tablet_bootstrap.cc:492] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: No bootstrap required, opened a new log
I20260927 10:50:12.368224 13720 ts_tablet_manager.cc:1397] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:50:12.368599 13719 tablet_bootstrap.cc:492] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9: No bootstrap required, opened a new log
I20260927 10:50:12.368695 13719 ts_tablet_manager.cc:1397] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:50:12.368686 13720 raft_consensus.cc:348] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } }
I20260927 10:50:12.368815 13723 tablet_bootstrap.cc:492] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Bootstrap starting.
I20260927 10:50:12.368824 13720 raft_consensus.cc:374] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:12.368865 13720 raft_consensus.cc:729] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f4e8905592544d89e0069f132cf39d5, State: Initialized, Role: FOLLOWER
I20260927 10:50:12.368997 13720 consensus_queue.cc:260] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } }
I20260927 10:50:12.369136 13719 raft_consensus.cc:348] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } }
I20260927 10:50:12.369241 13719 raft_consensus.cc:374] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:12.369277 13719 raft_consensus.cc:729] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5acccd0f20d448e7b4414693d2073ca9, State: Initialized, Role: FOLLOWER
I20260927 10:50:12.369305 13720 ts_tablet_manager.cc:1428] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:12.369407 13719 consensus_queue.cc:260] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } }
I20260927 10:50:12.370357 13723 tablet_bootstrap.cc:654] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:12.370818 13719 ts_tablet_manager.cc:1428] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:50:12.371886 13723 tablet_bootstrap.cc:492] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: No bootstrap required, opened a new log
I20260927 10:50:12.371975 13723 ts_tablet_manager.cc:1397] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:50:12.372485 13723 raft_consensus.cc:348] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } }
I20260927 10:50:12.372604 13723 raft_consensus.cc:374] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:12.372637 13723 raft_consensus.cc:729] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2bdfdc5bd449a1a91d7b32fe3ab147, State: Initialized, Role: FOLLOWER
I20260927 10:50:12.372836 13481 heartbeater.cc:503] Master 127.6.31.188:45305 was elected leader, sending a full tablet report...
I20260927 10:50:12.373291 13723 consensus_queue.cc:260] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } }
I20260927 10:50:12.373636 13723 ts_tablet_manager.cc:1428] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:50:12.374248 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.369745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:42299)
0927 10:50:12.371001 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.374112 (+ 3111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1184,"thread_start_us":91,"threads_started":1}
I20260927 10:50:12.374651 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.372070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.129:48169)
0927 10:50:12.374480 (+ 2410us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.374543 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2339,"thread_start_us":92,"threads_started":1}
I20260927 10:50:12.375293 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.374050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.131:38263)
0927 10:50:12.375152 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.375208 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1017,"thread_start_us":102,"threads_started":1}
I20260927 10:50:12.375413 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.374711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:53995)
0927 10:50:12.375204 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.375257 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
I20260927 10:50:12.376075 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.375664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.131:52003)
0927 10:50:12.375904 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.375968 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260927 10:50:12.376148 13617 heartbeater.cc:503] Master 127.6.31.188:45305 was elected leader, sending a full tablet report...
I20260927 10:50:12.376353 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.375803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.129:43959)
0927 10:50:12.376246 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.376296 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361}
W20260927 10:50:12.428764 13484 tablet.cc:2369] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:50:12.501063 13724 raft_consensus.cc:482] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:12.501240 13724 raft_consensus.cc:504] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } }
I20260927 10:50:12.502406 13724 leader_election.cc:288] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c2bdfdc5bd449a1a91d7b32fe3ab147 (127.6.31.131:38823), 5acccd0f20d448e7b4414693d2073ca9 (127.6.31.129:36455)
I20260927 10:50:12.507212 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.502634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38823 (local address 127.6.31.130:59431)
0927 10:50:12.502788 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.502806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:12.502949 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:12.503264 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:12.503268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:12.503288 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:12.503434 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.503444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.504453 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.504455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.505822 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.505832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.506695 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.506698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.506822 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.506845 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:50:12.506905 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 10:50:12.506994 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":40}
I20260927 10:50:12.507227 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.502751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:59431 (local address 127.6.31.131:38823)
0927 10:50:12.502902 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:12.502907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:12.502925 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:50:12.503061 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:12.503064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:12.503165 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:12.503240 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:12.503540 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.504345 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.505946 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.506561 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.506696 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.506760 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:12.506944 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 10:50:12.507023 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:50:12.507258 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.503055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:59083 (local address 127.6.31.129:36455)
0927 10:50:12.503242 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:12.503247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:12.503382 (+ 135us) server_negotiation.cc:407] Connection header received
0927 10:50:12.503466 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:12.503468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:12.503539 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:12.503617 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:12.503831 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.504530 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.505986 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.506580 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.506708 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.506761 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:12.506998 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 10:50:12.507101 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:50:12.507239 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.502931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36455 (local address 127.6.31.130:59083)
0927 10:50:12.503285 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.503298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:12.503386 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:12.503634 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:12.503636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:12.503650 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:12.503731 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.503735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.504621 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.504624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.505821 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:12.505832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.506740 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.506744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:12.506851 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.506875 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:50:12.506969 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:50:12.507034 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":38,"thread_start_us":153,"threads_started":1}
I20260927 10:50:12.507776 13298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077b25ac79145edb3a1673b8ceac691" candidate_uuid: "5f4e8905592544d89e0069f132cf39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5acccd0f20d448e7b4414693d2073ca9" is_pre_election: true
I20260927 10:50:12.507807 13571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077b25ac79145edb3a1673b8ceac691" candidate_uuid: "5f4e8905592544d89e0069f132cf39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" is_pre_election: true
I20260927 10:50:12.508000 13298 raft_consensus.cc:2454] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f4e8905592544d89e0069f132cf39d5 in term 0.
I20260927 10:50:12.508073 13571 raft_consensus.cc:2454] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f4e8905592544d89e0069f132cf39d5 in term 0.
I20260927 10:50:12.508401 13370 leader_election.cc:302] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5acccd0f20d448e7b4414693d2073ca9, 5f4e8905592544d89e0069f132cf39d5; no voters:
I20260927 10:50:12.508637 13724 raft_consensus.cc:2781] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:12.508697 13724 raft_consensus.cc:482] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:12.508733 13724 raft_consensus.cc:3037] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:12.509752 13724 raft_consensus.cc:504] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } }
I20260927 10:50:12.510084 13724 leader_election.cc:288] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [CANDIDATE]: Term 1 election: Requested vote from peers 4c2bdfdc5bd449a1a91d7b32fe3ab147 (127.6.31.131:38823), 5acccd0f20d448e7b4414693d2073ca9 (127.6.31.129:36455)
I20260927 10:50:12.510429 13298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077b25ac79145edb3a1673b8ceac691" candidate_uuid: "5f4e8905592544d89e0069f132cf39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5acccd0f20d448e7b4414693d2073ca9"
I20260927 10:50:12.510524 13298 raft_consensus.cc:3037] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:12.511251 13571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077b25ac79145edb3a1673b8ceac691" candidate_uuid: "5f4e8905592544d89e0069f132cf39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147"
I20260927 10:50:12.511382 13571 raft_consensus.cc:3037] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:12.511499 13298 raft_consensus.cc:2454] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f4e8905592544d89e0069f132cf39d5 in term 1.
I20260927 10:50:12.511770 13370 leader_election.cc:302] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5acccd0f20d448e7b4414693d2073ca9, 5f4e8905592544d89e0069f132cf39d5; no voters:
I20260927 10:50:12.511922 13724 raft_consensus.cc:2781] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:12.512159 13724 raft_consensus.cc:686] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 1 LEADER]: Becoming Leader. State: Replica: 5f4e8905592544d89e0069f132cf39d5, State: Running, Role: LEADER
I20260927 10:50:12.512454 13724 consensus_queue.cc:237] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } }
I20260927 10:50:12.512584 13571 raft_consensus.cc:2454] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f4e8905592544d89e0069f132cf39d5 in term 1.
I20260927 10:50:12.513159 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.512847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:55253)
0927 10:50:12.513022 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.513080 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 10:50:12.514823 13165 catalog_manager.cc:5495] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f4e8905592544d89e0069f132cf39d5 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "5f4e8905592544d89e0069f132cf39d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4e8905592544d89e0069f132cf39d5" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 38551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 } health_report { overall_health: UNKNOWN } } }
I20260927 10:50:12.551527 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.548173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.6.31.130:38551)
0927 10:50:12.548299 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:12.548303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:12.548332 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:50:12.548488 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:12.548490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:12.548544 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:12.548645 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:12.548831 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.549344 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.550176 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:12.550384 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:12.550465 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:12.550677 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:12.551166 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:12.551251 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:12.551316 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 10:50:12.551362 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:50:12.554308 13298 raft_consensus.cc:1261] T 7077b25ac79145edb3a1673b8ceac691 P 5acccd0f20d448e7b4414693d2073ca9 [term 1 FOLLOWER]: Refusing update from remote peer 5f4e8905592544d89e0069f132cf39d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:12.554616 13571 raft_consensus.cc:1261] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 1 FOLLOWER]: Refusing update from remote peer 5f4e8905592544d89e0069f132cf39d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:12.554840 13724 consensus_queue.cc:1035] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5acccd0f20d448e7b4414693d2073ca9" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:12.555202 13733 consensus_queue.cc:1035] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c2bdfdc5bd449a1a91d7b32fe3ab147" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:12.555948 13344 heartbeater.cc:503] Master 127.6.31.188:45305 was elected leader, sending a full tablet report...
I20260927 10:50:12.556658 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.556376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.129:46149)
0927 10:50:12.556521 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.556587 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 10:50:12.557721 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.557444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.131:36021)
0927 10:50:12.557583 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.557651 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 10:50:12.559356 13736 mvcc.cc:205] Tried to move back new op lower bound from 7333913446617358336 to 7333913446453620736. Current Snapshot: MvccSnapshot[applied={T|T < 7333913446617358336}]
I20260927 10:50:12.559684 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.559389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:34119)
0927 10:50:12.559534 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.559606 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 10:50:12.561611 13739 mvcc.cc:205] Tried to move back new op lower bound from 7333913446617358336 to 7333913446453620736. Current Snapshot: MvccSnapshot[applied={T|T < 7333913446617358336 or (T in {7333913446617358336})}]
I20260927 10:50:12.561700 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.561440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.130:54929)
0927 10:50:12.561575 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.561626 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 10:50:12.563828 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.563161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.131:33773)
0927 10:50:12.563698 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.563754 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456}
I20260927 10:50:12.564419 13737 mvcc.cc:205] Tried to move back new op lower bound from 7333913446617358336 to 7333913446453620736. Current Snapshot: MvccSnapshot[applied={T|T < 7333913446617358336}]
I20260927 10:50:12.567612 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.567334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36259 (local address 127.6.31.129:34635)
0927 10:50:12.567481 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.567537 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 10:50:12.568041 13165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:12.568178 13165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:12.571993 13165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e575508cb3934882849930d9b24ed899: Sending DeleteTablet for 3 replicas of tablet 7077b25ac79145edb3a1673b8ceac691
I20260927 10:50:12.572715 13415 tablet_service.cc:1514] Processing DeleteTablet for tablet 7077b25ac79145edb3a1673b8ceac691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:12 UTC) from {username='oryx', principal='oryx/127.6.31.188@KRBTEST.COM'} at 127.0.0.1:58982
I20260927 10:50:12.573328 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 13125
I20260927 10:50:12.574198 13551 tablet_service.cc:1514] Processing DeleteTablet for tablet 7077b25ac79145edb3a1673b8ceac691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:12 UTC) from {username='oryx', principal='oryx/127.6.31.188@KRBTEST.COM'} at 127.0.0.1:45780
I20260927 10:50:12.574584 13747 tablet_replica.cc:330] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: stopping tablet replica
I20260927 10:50:12.574816 13747 raft_consensus.cc:2229] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:12.574975 13747 raft_consensus.cc:2258] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:12.576385 13746 tablet_replica.cc:330] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: stopping tablet replica
I20260927 10:50:12.576535 13746 raft_consensus.cc:2229] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:50:12.576661 13747 ts_tablet_manager.cc:1905] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:12.576885 13746 raft_consensus.cc:2258] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:12.577622 13746 ts_tablet_manager.cc:1905] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:12.579658 13747 ts_tablet_manager.cc:1918] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:12.579748 13747 log.cc:1196] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/7077b25ac79145edb3a1673b8ceac691
I20260927 10:50:12.580116 13747 ts_tablet_manager.cc:1939] T 7077b25ac79145edb3a1673b8ceac691 P 4c2bdfdc5bd449a1a91d7b32fe3ab147: Deleting consensus metadata
I20260927 10:50:12.580865 13746 ts_tablet_manager.cc:1918] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:12.580940 13746 log.cc:1196] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/7077b25ac79145edb3a1673b8ceac691
I20260927 10:50:12.581014 13151 catalog_manager.cc:4841] TS 4c2bdfdc5bd449a1a91d7b32fe3ab147 (127.6.31.131:38823): tablet 7077b25ac79145edb3a1673b8ceac691 (table test-table [id=cf34393bdb7d45f5990ff248e2a118e3]) successfully deleted
I20260927 10:50:12.581210 13746 ts_tablet_manager.cc:1939] T 7077b25ac79145edb3a1673b8ceac691 P 5f4e8905592544d89e0069f132cf39d5: Deleting consensus metadata
I20260927 10:50:12.581936 13152 catalog_manager.cc:4841] TS 5f4e8905592544d89e0069f132cf39d5 (127.6.31.130:38551): tablet 7077b25ac79145edb3a1673b8ceac691 (table test-table [id=cf34393bdb7d45f5990ff248e2a118e3]) successfully deleted
W20260927 10:50:12.582479 13151 connection.cc:535] client connection to 127.6.31.129:36455 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:50:12.583227 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 13352
I20260927 10:50:12.583386 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:12.583124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36455 (local address 127.0.0.1:49284)
0927 10:50:12.583238 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:12.583291 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 10:50:12.589164 13151 catalog_manager.cc:4584] TS 5acccd0f20d448e7b4414693d2073ca9 (127.6.31.129:36455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7077b25ac79145edb3a1673b8ceac691: Network error: Client connection negotiation failed: client connection to 127.6.31.129:36455: connect: Connection refused (error 111)
I20260927 10:50:12.589380 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 13488
I20260927 10:50:12.595956 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 12838
I20260927 10:50:12.601709 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 12951
2026-09-27T10:50:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7276 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:50:12 dist-test-slave-c60x krb5kdc[13750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:50:12 dist-test-slave-c60x krb5kdc[13750](info): set up 2 sockets
Sep 27 10:50:12 dist-test-slave-c60x krb5kdc[13750](info): commencing operation
krb5kdc: starting...
W20260927 10:50:13.655725 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:50:13 dist-test-slave-c60x krb5kdc[13750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.190.keytab.
Entry for principal oryx/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.190.keytab.
I20260927 10:50:13.782979 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:43083
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.190.keytab
--principal=oryx/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:43083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:13.892522 13763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:13.892824 13763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:13.892895 13763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:13.896935 13763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:13.897013 13763 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:50:13.897043 13763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:13.897090 13763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:13.897140 13763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:13.897183 13763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:13.903357 13763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:43083
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.190.keytab
--principal=oryx/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:43083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:13.904517 13763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:13.905553 13763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:13.914196 13768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:13.914209 13771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:13.914255 13763 server_base.cc:1034] running on GCE node
W20260927 10:50:13.914285 13769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:13.915140 13763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:13.916291 13763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:13.917469 13763 hybrid_clock.cc:648] HybridClock initialized: now 1790506213917460 us; error 30 us; skew 500 ppm
Sep 27 10:50:13 dist-test-slave-c60x krb5kdc[13750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506213, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:13.920221 13763 init.cc:377] Logged in from keytab as oryx/127.6.31.190@KRBTEST.COM (short username oryx)
I20260927 10:50:13.921108 13763 webserver.cc:460] Webserver started at http://127.6.31.190:37977/ using document root <none> and password file <none>
I20260927 10:50:13.921656 13763 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:13.921742 13763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:13.921962 13763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:13.923998 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "8e8013458fef47058ef34dbd1890385e"
format_stamp: "Formatted at 2026-09-27 10:50:13 on dist-test-slave-c60x"
server_key: "2778ce299360e56f281184a31caf0221"
server_key_iv: "582db36f50b57c198b03eb807045bd5c"
server_key_version: "encryptionkey@0"
I20260927 10:50:13.924546 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "8e8013458fef47058ef34dbd1890385e"
format_stamp: "Formatted at 2026-09-27 10:50:13 on dist-test-slave-c60x"
server_key: "2778ce299360e56f281184a31caf0221"
server_key_iv: "582db36f50b57c198b03eb807045bd5c"
server_key_version: "encryptionkey@0"
I20260927 10:50:13.928129 13763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:13.930380 13778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:13.931545 13763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:13.931681 13763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "8e8013458fef47058ef34dbd1890385e"
format_stamp: "Formatted at 2026-09-27 10:50:13 on dist-test-slave-c60x"
server_key: "2778ce299360e56f281184a31caf0221"
server_key_iv: "582db36f50b57c198b03eb807045bd5c"
server_key_version: "encryptionkey@0"
I20260927 10:50:13.931788 13763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:13.953493 13763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:13.956619 13763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:13.956844 13763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:13.965445 13763 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:43083
I20260927 10:50:13.965481 13840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:43083 every 8 connection(s)
I20260927 10:50:13.966478 13763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:13.969354 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 13763
I20260927 10:50:13.969441 13841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:13.969513 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:13.969805 6270 external_mini_cluster.cc:1428] Setting key 0d52e403b94acf45023bae893685280b
I20260927 10:50:13.976527 13841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e: Bootstrap starting.
Sep 27 10:50:13 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:13.979806 13841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:13.980650 13841 log.cc:824] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:13.983331 13841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e: No bootstrap required, opened a new log
I20260927 10:50:13.986606 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:13.971220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.6.31.190:43083)
0927 10:50:13.971637 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:13.971655 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:13.971703 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:50:13.972481 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:13.972499 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:13.972835 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:13.973112 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:13.974057 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:13.975072 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:13.976314 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:13.976884 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:13.977016 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:13.980249 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:13.980254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:13.980270 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:13.980300 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:13.983063 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:13.983821 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:13.983826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:13.983833 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:13.983975 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:13.984300 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:13.984303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:13.984305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:13.984813 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:13.984988 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:13.985296 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 10:50:13.985609 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":148,"threads_started":1}
I20260927 10:50:13.987378 13841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8013458fef47058ef34dbd1890385e" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43083 } }
I20260927 10:50:13.987901 13841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:13.988004 13841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e8013458fef47058ef34dbd1890385e, State: Initialized, Role: FOLLOWER
I20260927 10:50:13.988605 13841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8013458fef47058ef34dbd1890385e" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43083 } }
I20260927 10:50:13.988751 13841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:13.988858 13841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:13.988956 13841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:13.989921 13841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8013458fef47058ef34dbd1890385e" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43083 } }
I20260927 10:50:13.990309 13841 leader_election.cc:302] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e8013458fef47058ef34dbd1890385e; no voters:
I20260927 10:50:13.990645 13841 leader_election.cc:288] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:13.990821 13846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:13.991065 13846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [term 1 LEADER]: Becoming Leader. State: Replica: 8e8013458fef47058ef34dbd1890385e, State: Running, Role: LEADER
I20260927 10:50:13.991425 13846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8013458fef47058ef34dbd1890385e" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43083 } }
I20260927 10:50:13.991907 13841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:13.993000 13847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e8013458fef47058ef34dbd1890385e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8013458fef47058ef34dbd1890385e" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43083 } } }
I20260927 10:50:13.993124 13847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:13.993419 13848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e8013458fef47058ef34dbd1890385e. Latest consensus state: current_term: 1 leader_uuid: "8e8013458fef47058ef34dbd1890385e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8013458fef47058ef34dbd1890385e" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43083 } } }
I20260927 10:50:13.993518 13848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:13.993755 13855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:13.996640 13855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:14.002940 13855 catalog_manager.cc:1347] Generated new cluster ID: 0dcb1b5acae74ca8a33debc29bea3e84
I20260927 10:50:14.003021 13855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:14.015633 13855 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:14.016637 13855 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:14.032471 13855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e: Generated new TSK 0
I20260927 10:50:14.033273 13855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab.
Entry for principal oryx/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab.
I20260927 10:50:14.070714 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:43083
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab
--principal=oryx/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:14.185626 13867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:14.185902 13867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:14.185948 13867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:14.190057 13867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:14.190130 13867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:14.190171 13867 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:50:14.190205 13867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:14.195736 13867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab
--principal=oryx/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.31.190:43083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:14.196889 13867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:14.197878 13867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:14.208039 13873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:14.208166 13867 server_base.cc:1034] running on GCE node
W20260927 10:50:14.208303 13872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:14.208039 13875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:14.209336 13867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:14.210043 13867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:14.211256 13867 hybrid_clock.cc:648] HybridClock initialized: now 1790506214211238 us; error 32 us; skew 500 ppm
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506214, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:14.214154 13867 init.cc:377] Logged in from keytab as oryx/127.6.31.129@KRBTEST.COM (short username oryx)
I20260927 10:50:14.214993 13867 webserver.cc:460] Webserver started at http://127.6.31.129:39955/ using document root <none> and password file <none>
I20260927 10:50:14.215567 13867 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:14.215636 13867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:14.215794 13867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:14.217777 13867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "f37bba543be6487ca802300c46bccaa7"
format_stamp: "Formatted at 2026-09-27 10:50:14 on dist-test-slave-c60x"
server_key: "4e888ad99210888eebfc0a30bec9ead2"
server_key_iv: "e62773315e5fbcdcbcc6424df6b8c609"
server_key_version: "encryptionkey@0"
I20260927 10:50:14.218276 13867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "f37bba543be6487ca802300c46bccaa7"
format_stamp: "Formatted at 2026-09-27 10:50:14 on dist-test-slave-c60x"
server_key: "4e888ad99210888eebfc0a30bec9ead2"
server_key_iv: "e62773315e5fbcdcbcc6424df6b8c609"
server_key_version: "encryptionkey@0"
I20260927 10:50:14.221953 13867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:50:14.224376 13882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:14.225714 13867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:50:14.225878 13867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "f37bba543be6487ca802300c46bccaa7"
format_stamp: "Formatted at 2026-09-27 10:50:14 on dist-test-slave-c60x"
server_key: "4e888ad99210888eebfc0a30bec9ead2"
server_key_iv: "e62773315e5fbcdcbcc6424df6b8c609"
server_key_version: "encryptionkey@0"
I20260927 10:50:14.226013 13867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:14.242888 13867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:14.246177 13867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:14.246409 13867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:14.248119 13867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:14.248195 13867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:14.248250 13867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:14.248271 13867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506214, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:14.259474 13867 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:45493
I20260927 10:50:14.259500 13997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:45493 every 8 connection(s)
I20260927 10:50:14.260541 13867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:14.262602 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.248920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:49263 (local address 127.6.31.190:43083)
0927 10:50:14.249061 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.249065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.250203 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 10:50:14.251208 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.251212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.251272 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.251335 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:14.252718 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.253440 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.254650 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.254899 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.255015 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.258625 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:14.258629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:14.258631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:14.258658 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:14.260332 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.260951 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.260955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.260956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.261048 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.261429 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.261432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.261456 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.261705 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:14.261800 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.262351 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.262479 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:50:14.263513 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.249460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43083 (local address 127.6.31.129:49263)
0927 10:50:14.249973 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.250020 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.250929 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.251739 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.251745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.252085 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:14.252568 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.252580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.253583 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.253591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.254525 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.254533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.255067 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.255071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.255260 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.256016 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:14.256041 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:14.258436 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:14.260453 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.260462 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.260478 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.260820 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.261154 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.261157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.261159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.261322 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.261840 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:14.261845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:14.262073 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.262267 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.262531 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":154,"threads_started":1}
I20260927 10:50:14.264993 13998 heartbeater.cc:347] Connected to a master server at 127.6.31.190:43083
I20260927 10:50:14.265151 13998 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:14.265710 13998 heartbeater.cc:511] Master 127.6.31.190:43083 requested a full tablet report, sending...
I20260927 10:50:14.267524 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 13867
I20260927 10:50:14.267534 13794 ts_manager.cc:194] Registered new tserver with Master: f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493)
I20260927 10:50:14.267658 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:50:14.267973 6270 external_mini_cluster.cc:1428] Setting key 64a2a0f3b83aa2a4c1d6201a94e3c0f8
I20260927 10:50:14.269111 13794 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:49263
I20260927 10:50:14.276535 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.267343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43083 (local address 127.6.31.129:39951)
0927 10:50:14.267618 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.267635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.267751 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.268842 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.268845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.269107 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:14.269244 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.269253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.270964 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.270966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.271594 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.271627 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.272117 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.272120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.272218 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.272773 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:14.272785 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:14.273114 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:14.274870 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.274874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.274877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.275128 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.275457 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.275459 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.275460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.275535 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.275979 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:14.275981 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:14.276068 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.276224 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.276339 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":118,"threads_started":1}
I20260927 10:50:14.276613 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.268475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:39951 (local address 127.6.31.190:43083)
0927 10:50:14.268580 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.268583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.268594 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:50:14.268624 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.268626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.268718 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.268803 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:14.269367 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.270024 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.271741 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.271986 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.272117 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.273302 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:14.273305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:14.273307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:14.273328 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:14.274758 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.275248 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.275252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.275254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.275356 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.275622 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.275628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.275633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.275877 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:14.275989 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.276315 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.276443 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
WARNING: no policy specified for oryx/127.6.31.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab.
Entry for principal oryx/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab.
I20260927 10:50:14.296684 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:43083
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab
--principal=oryx/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:14.411581 14007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:14.411847 14007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:14.411926 14007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:14.415985 14007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:14.416059 14007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:14.416098 14007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:50:14.416164 14007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:14.421475 14007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab
--principal=oryx/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.31.190:43083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:14.422590 14007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:14.423691 14007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:14.433557 14012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:14.433523 14013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:14.433544 14015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:14.433799 14007 server_base.cc:1034] running on GCE node
I20260927 10:50:14.434235 14007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:14.434870 14007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:14.436079 14007 hybrid_clock.cc:648] HybridClock initialized: now 1790506214436079 us; error 63 us; skew 500 ppm
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506214, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:14.438980 14007 init.cc:377] Logged in from keytab as oryx/127.6.31.130@KRBTEST.COM (short username oryx)
I20260927 10:50:14.440140 14007 webserver.cc:460] Webserver started at http://127.6.31.130:43761/ using document root <none> and password file <none>
I20260927 10:50:14.440727 14007 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:14.440785 14007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:14.441002 14007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:14.443048 14007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "7d55abc064024922946718f3426ff476"
format_stamp: "Formatted at 2026-09-27 10:50:14 on dist-test-slave-c60x"
server_key: "ccef5d123e4e5ed377fd81894c080582"
server_key_iv: "81c5fb5cdadfd372cb6784e6d62a07c1"
server_key_version: "encryptionkey@0"
I20260927 10:50:14.443689 14007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "7d55abc064024922946718f3426ff476"
format_stamp: "Formatted at 2026-09-27 10:50:14 on dist-test-slave-c60x"
server_key: "ccef5d123e4e5ed377fd81894c080582"
server_key_iv: "81c5fb5cdadfd372cb6784e6d62a07c1"
server_key_version: "encryptionkey@0"
I20260927 10:50:14.447381 14007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:14.449765 14022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:14.450971 14007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:50:14.451090 14007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "7d55abc064024922946718f3426ff476"
format_stamp: "Formatted at 2026-09-27 10:50:14 on dist-test-slave-c60x"
server_key: "ccef5d123e4e5ed377fd81894c080582"
server_key_iv: "81c5fb5cdadfd372cb6784e6d62a07c1"
server_key_version: "encryptionkey@0"
I20260927 10:50:14.451243 14007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:14.464193 14007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:14.467985 14007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:14.468199 14007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:14.469645 14007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:14.469722 14007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:14.469786 14007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:14.469837 14007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506214, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:14.480878 14007 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:34129
I20260927 10:50:14.480916 14137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:34129 every 8 connection(s)
I20260927 10:50:14.482028 14007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:14.484447 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.470404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:42917 (local address 127.6.31.190:43083)
0927 10:50:14.470564 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.470571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.471483 (+ 912us) server_negotiation.cc:407] Connection header received
0927 10:50:14.472742 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.472748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.472816 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.472881 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:14.474716 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.475398 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.476651 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.477014 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.477125 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.480321 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:14.480324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:14.480326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:14.480350 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:14.482055 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.482814 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.482818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.482820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.482922 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.483317 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.483320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.483321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.483505 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:14.483620 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.484202 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.484322 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:50:14.485283 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.470820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43083 (local address 127.6.31.130:42917)
0927 10:50:14.471314 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.471368 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.472476 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.473343 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.473354 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.473827 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:14.474542 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.474565 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.475535 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.475544 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.476529 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.476538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.477313 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.477316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.477457 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.478064 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:14.478090 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:14.480139 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:14.482295 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.482301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.482316 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.482681 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.483013 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.483015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.483017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.483225 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.483662 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:14.483668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:14.483908 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.484073 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.484349 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":104,"threads_started":1}
I20260927 10:50:14.486505 14138 heartbeater.cc:347] Connected to a master server at 127.6.31.190:43083
I20260927 10:50:14.486671 14138 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:14.487239 14138 heartbeater.cc:511] Master 127.6.31.190:43083 requested a full tablet report, sending...
I20260927 10:50:14.487378 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 14007
I20260927 10:50:14.487526 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:50:14.487847 6270 external_mini_cluster.cc:1428] Setting key e6c57738146474f95dd7aba366222fa8
I20260927 10:50:14.488343 13794 ts_manager.cc:194] Registered new tserver with Master: 7d55abc064024922946718f3426ff476 (127.6.31.130:34129)
I20260927 10:50:14.489198 13794 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:42917
I20260927 10:50:14.497622 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.488946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:54243 (local address 127.6.31.190:43083)
0927 10:50:14.489066 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.489070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.489277 (+ 207us) server_negotiation.cc:407] Connection header received
0927 10:50:14.489538 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.489541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.489603 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.489683 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:14.490290 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.491131 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.492123 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.492342 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.492453 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.493842 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:14.493846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:14.493849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:14.493878 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:14.495757 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.496342 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.496344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.496346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.496421 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.496727 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.496729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.496730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.496941 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:14.497020 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.497388 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.497488 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:50:14.497726 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.488903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43083 (local address 127.6.31.130:54243)
0927 10:50:14.489160 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.489176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.489349 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.489733 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.489736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.489979 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:14.490131 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.490142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.491325 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.491327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.492008 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.492016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.492457 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.492461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.492565 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.493184 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:14.493204 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:14.493678 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:14.495892 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.495897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.495901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.496225 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.496513 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.496516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.496518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.496631 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.497040 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:14.497043 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:14.497164 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.497312 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.497497 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for oryx/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab.
Entry for principal oryx/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab.
I20260927 10:50:14.518664 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:43083
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab
--principal=oryx/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:14.629041 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:14.629381 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:14.629468 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:14.633605 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:14.633713 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:14.633774 14147 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:50:14.633811 14147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:14.639185 14147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab
--principal=oryx/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.31.190:43083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:14.640161 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:14.641160 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:14.650835 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:14.650840 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:14.650903 14147 server_base.cc:1034] running on GCE node
W20260927 10:50:14.651119 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:14.651520 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:14.652158 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:14.653332 14147 hybrid_clock.cc:648] HybridClock initialized: now 1790506214653320 us; error 25 us; skew 500 ppm
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506214, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:14.656136 14147 init.cc:377] Logged in from keytab as oryx/127.6.31.131@KRBTEST.COM (short username oryx)
I20260927 10:50:14.656937 14147 webserver.cc:460] Webserver started at http://127.6.31.131:37613/ using document root <none> and password file <none>
I20260927 10:50:14.657475 14147 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:14.657562 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:14.657790 14147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:14.659797 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69"
format_stamp: "Formatted at 2026-09-27 10:50:14 on dist-test-slave-c60x"
server_key: "2df9690cdc0128a6aac66e78e2d8a934"
server_key_iv: "2df5fb51cb9e3a0fa855078c34f7a027"
server_key_version: "encryptionkey@0"
I20260927 10:50:14.660344 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69"
format_stamp: "Formatted at 2026-09-27 10:50:14 on dist-test-slave-c60x"
server_key: "2df9690cdc0128a6aac66e78e2d8a934"
server_key_iv: "2df5fb51cb9e3a0fa855078c34f7a027"
server_key_version: "encryptionkey@0"
I20260927 10:50:14.663877 14147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:14.666292 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:14.667419 14147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:14.667541 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69"
format_stamp: "Formatted at 2026-09-27 10:50:14 on dist-test-slave-c60x"
server_key: "2df9690cdc0128a6aac66e78e2d8a934"
server_key_iv: "2df5fb51cb9e3a0fa855078c34f7a027"
server_key_version: "encryptionkey@0"
I20260927 10:50:14.667640 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:14.677706 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:14.680974 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:14.681170 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:14.682524 14147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:14.682595 14147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:14.682662 14147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:14.682709 14147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506214, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:14.694093 14147 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:41865
I20260927 10:50:14.694166 14277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:41865 every 8 connection(s)
I20260927 10:50:14.695159 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:50:14.696578 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.683600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:49357 (local address 127.6.31.190:43083)
0927 10:50:14.683730 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.683734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.684777 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 10:50:14.685899 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.685903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.685963 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.686071 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:14.687311 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.688074 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.689302 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.689538 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.689649 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.692765 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:14.692771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:14.692776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:14.692806 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:14.694507 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.695001 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.695005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.695007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.695118 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.695490 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.695494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.695495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.695727 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:14.695834 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.696362 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.696461 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:50:14.697407 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.684129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43083 (local address 127.6.31.131:49357)
0927 10:50:14.684632 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.684671 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.685676 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.686291 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.686297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.686599 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:14.687163 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.687193 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.688202 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.688205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.689171 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.689187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.689699 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.689703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.689918 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.690551 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:14.690570 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:14.692592 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:14.694631 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.694636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.694649 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.694901 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.695243 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.695247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.695251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.695394 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.695888 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:14.695896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:14.696133 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.696292 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.696524 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":138,"threads_started":1}
I20260927 10:50:14.698679 14278 heartbeater.cc:347] Connected to a master server at 127.6.31.190:43083
I20260927 10:50:14.698833 14278 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:14.699376 14278 heartbeater.cc:511] Master 127.6.31.190:43083 requested a full tablet report, sending...
I20260927 10:50:14.700423 13795 ts_manager.cc:194] Registered new tserver with Master: c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865)
I20260927 10:50:14.701189 13795 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:49357
I20260927 10:50:14.705034 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 14147
I20260927 10:50:14.705174 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:50:14.705859 6270 external_mini_cluster.cc:1428] Setting key 07d34326f62b028c80ec4452c8f2831e
I20260927 10:50:14.707933 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:50:14.708444 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.700912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:47193 (local address 127.6.31.190:43083)
0927 10:50:14.701031 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.701035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.701298 (+ 263us) server_negotiation.cc:407] Connection header received
0927 10:50:14.701382 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.701385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.701438 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.701516 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:14.702028 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.702778 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.703624 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.703831 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.703910 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.704912 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:14.704916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:14.704919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:14.704946 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:14.706833 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.707330 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.707334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.707335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.707418 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.707680 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.707704 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.707707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.707911 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:14.708026 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.708258 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.708341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:50:14.708549 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.700795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43083 (local address 127.6.31.131:47193)
0927 10:50:14.701171 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.701195 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.701293 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.701556 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.701559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.701794 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:14.701901 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.701908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.702904 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.702906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.703526 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.703533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.703926 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.703928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.703993 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.704429 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:14.704440 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:14.704765 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:14.706950 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.706952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.706953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.707243 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.707527 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:14.707529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:14.707530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:14.707600 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:14.708021 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:14.708024 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:14.708109 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.708218 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.708340 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":103,"threads_started":1}
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506214, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:50:14.732030 13844 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:14.725834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.6.31.190:43083)
0927 10:50:14.725965 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.725969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.725980 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:50:14.726019 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.726022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.726059 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.726148 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:14.726844 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.727421 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.728247 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.728460 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.728565 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.731876 (+ 3311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54350: BlockingRecv error: recv got EOF from 127.0.0.1:54350 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 10:50:14 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506214, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:14.741981 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.733555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.6.31.190:43083)
0927 10:50:14.733689 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.733694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.733818 (+ 124us) server_negotiation.cc:407] Connection header received
0927 10:50:14.733966 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.733969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.734010 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.734101 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:14.734675 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.735265 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.736153 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.736439 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.736546 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.738875 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:14.738878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:14.738880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:14.738902 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:14.740385 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.740788 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.740791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.740792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.740868 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:14.741292 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:14.741295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:14.741296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:14.741467 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:14.741573 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.741760 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.741860 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:50:14.744836 13795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:50:14.747875 13795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:14.761735 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.756685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.6.31.131:41865)
0927 10:50:14.757151 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.757161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.757183 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:50:14.757256 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.757263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.757447 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.757828 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:14.757955 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.758723 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.760550 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.761067 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.761204 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.761380 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.761486 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.761577 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":106,"threads_started":1}
I20260927 10:50:14.761725 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.756233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:41865 (local address 127.0.0.1:50958)
0927 10:50:14.756624 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.756638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.756736 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.757587 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.757590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.757603 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:14.757693 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.757701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.758830 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.758836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.760401 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.760412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.761194 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.761198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.761303 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.761339 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.761458 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.761578 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":67,"thread_start_us":84,"threads_started":1}
I20260927 10:50:14.762019 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.756999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.6.31.129:45493)
0927 10:50:14.757255 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.757268 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.757529 (+ 261us) server_negotiation.cc:407] Connection header received
0927 10:50:14.757603 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.757607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.757775 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.757923 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:14.758134 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.759221 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.760380 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.761157 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.761262 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.761431 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.761790 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.761886 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260927 10:50:14.762108 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.755935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34129 (local address 127.0.0.1:52092)
0927 10:50:14.756238 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.756282 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.756395 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.757455 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.757461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.757483 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:14.757644 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.757651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.759106 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.759109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.760407 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.760412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.761386 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.761389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.761484 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.761528 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.761576 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.761659 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":82,"thread_start_us":98,"threads_started":1}
I20260927 10:50:14.762390 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.756067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.6.31.130:34129)
0927 10:50:14.756972 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.756984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.757005 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:50:14.757080 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.757088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.757296 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.757454 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:14.757756 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.758969 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.760517 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.761259 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.761883 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.762037 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.762086 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.762254 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":141,"threads_started":1}
I20260927 10:50:14.762526 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.756604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:45493 (local address 127.0.0.1:56492)
0927 10:50:14.757354 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.757368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.757447 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.757913 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.757916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.757933 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:14.758021 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.758026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.759340 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.759343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.760257 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.760265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.761481 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.761484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.761566 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.761588 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.761645 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.761704 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260927 10:50:14.764801 14072 tablet_service.cc:1467] Processing CreateTablet for tablet 07ecebd5affa4684bb9d58a5b9706eeb (DEFAULT_TABLE table=test-table [id=2a95123a7fa347a6a0410f854873ca26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:14.764801 14212 tablet_service.cc:1467] Processing CreateTablet for tablet 07ecebd5affa4684bb9d58a5b9706eeb (DEFAULT_TABLE table=test-table [id=2a95123a7fa347a6a0410f854873ca26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:14.765770 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ecebd5affa4684bb9d58a5b9706eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:14.765810 14072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ecebd5affa4684bb9d58a5b9706eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:14.766758 13932 tablet_service.cc:1467] Processing CreateTablet for tablet 07ecebd5affa4684bb9d58a5b9706eeb (DEFAULT_TABLE table=test-table [id=2a95123a7fa347a6a0410f854873ca26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:14.767691 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ecebd5affa4684bb9d58a5b9706eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:14.772495 14306 tablet_bootstrap.cc:492] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: Bootstrap starting.
I20260927 10:50:14.774613 14308 tablet_bootstrap.cc:492] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: Bootstrap starting.
I20260927 10:50:14.774662 14306 tablet_bootstrap.cc:654] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:14.775429 14307 tablet_bootstrap.cc:492] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: Bootstrap starting.
I20260927 10:50:14.775470 14306 log.cc:824] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:14.777094 14308 tablet_bootstrap.cc:654] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:14.777351 14306 tablet_bootstrap.cc:492] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: No bootstrap required, opened a new log
I20260927 10:50:14.777575 14306 ts_tablet_manager.cc:1397] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:50:14.777706 14307 tablet_bootstrap.cc:654] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:14.777959 14308 log.cc:824] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:14.778441 14307 log.cc:824] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:14.779883 14308 tablet_bootstrap.cc:492] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: No bootstrap required, opened a new log
I20260927 10:50:14.780068 14308 ts_tablet_manager.cc:1397] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:50:14.780395 14307 tablet_bootstrap.cc:492] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: No bootstrap required, opened a new log
I20260927 10:50:14.780557 14307 ts_tablet_manager.cc:1397] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:50:14.781309 14306 raft_consensus.cc:348] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.781539 14306 raft_consensus.cc:374] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:14.781611 14306 raft_consensus.cc:729] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ddb9d52acd4afb8c1d75e3be65cf69, State: Initialized, Role: FOLLOWER
I20260927 10:50:14.782064 14306 consensus_queue.cc:260] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.782680 14278 heartbeater.cc:503] Master 127.6.31.190:43083 was elected leader, sending a full tablet report...
I20260927 10:50:14.783034 14306 ts_tablet_manager.cc:1428] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:50:14.783795 14308 raft_consensus.cc:348] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.784044 14308 raft_consensus.cc:374] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:14.784101 14308 raft_consensus.cc:729] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37bba543be6487ca802300c46bccaa7, State: Initialized, Role: FOLLOWER
I20260927 10:50:14.784494 14308 consensus_queue.cc:260] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.785054 13998 heartbeater.cc:503] Master 127.6.31.190:43083 was elected leader, sending a full tablet report...
I20260927 10:50:14.785411 14308 ts_tablet_manager.cc:1428] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 10:50:14.786105 14307 raft_consensus.cc:348] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.786394 14307 raft_consensus.cc:374] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:14.786492 14307 raft_consensus.cc:729] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d55abc064024922946718f3426ff476, State: Initialized, Role: FOLLOWER
I20260927 10:50:14.786896 14307 consensus_queue.cc:260] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.787528 14138 heartbeater.cc:503] Master 127.6.31.190:43083 was elected leader, sending a full tablet report...
I20260927 10:50:14.787890 14307 ts_tablet_manager.cc:1428] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260927 10:50:14.814834 14313 raft_consensus.cc:482] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:14.815027 14313 raft_consensus.cc:504] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.816331 14313 leader_election.cc:288] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d55abc064024922946718f3426ff476 (127.6.31.130:34129), c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865)
I20260927 10:50:14.821821 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.816892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:34975 (local address 127.6.31.131:41865)
0927 10:50:14.817042 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.817047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.817268 (+ 221us) server_negotiation.cc:407] Connection header received
0927 10:50:14.817358 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.817361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.817413 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.817488 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:14.817777 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.818586 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.820496 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.821131 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.821248 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.821302 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.821615 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.821687 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:50:14.821821 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.816702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:54155 (local address 127.6.31.130:34129)
0927 10:50:14.816870 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.816877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.817119 (+ 242us) server_negotiation.cc:407] Connection header received
0927 10:50:14.817366 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.817370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.817420 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.817470 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:14.817880 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.818614 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.820496 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.821116 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.821256 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.821302 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.821615 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.821687 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 10:50:14.821897 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.816550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34129 (local address 127.6.31.129:54155)
0927 10:50:14.817032 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.817046 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.817211 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.817619 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.817621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.817655 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:14.817769 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.817775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.818726 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.818729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.820342 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.820353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.821289 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.821294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.821454 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.821505 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.821585 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.821736 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":302,"thread_start_us":105,"threads_started":1}
I20260927 10:50:14.821897 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.816637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:41865 (local address 127.6.31.129:34975)
0927 10:50:14.817151 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:14.817165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:14.817270 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:14.817501 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:14.817504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:14.817522 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:14.817652 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.817659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.818735 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.818738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.820342 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:14.820353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.821289 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.821294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:14.821482 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.821519 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 10:50:14.821585 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:50:14.821706 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":228,"thread_start_us":132,"threads_started":1}
I20260927 10:50:14.822538 14092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ecebd5affa4684bb9d58a5b9706eeb" candidate_uuid: "f37bba543be6487ca802300c46bccaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d55abc064024922946718f3426ff476" is_pre_election: true
I20260927 10:50:14.822609 14232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ecebd5affa4684bb9d58a5b9706eeb" candidate_uuid: "f37bba543be6487ca802300c46bccaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" is_pre_election: true
I20260927 10:50:14.822872 14092 raft_consensus.cc:2454] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f37bba543be6487ca802300c46bccaa7 in term 0.
I20260927 10:50:14.822872 14232 raft_consensus.cc:2454] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f37bba543be6487ca802300c46bccaa7 in term 0.
I20260927 10:50:14.823421 13883 leader_election.cc:302] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, c4ddb9d52acd4afb8c1d75e3be65cf69, f37bba543be6487ca802300c46bccaa7; no voters:
I20260927 10:50:14.823701 14313 raft_consensus.cc:2781] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:14.823812 14313 raft_consensus.cc:482] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:14.823858 14313 raft_consensus.cc:3037] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:14.824874 14313 raft_consensus.cc:504] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.825222 14313 leader_election.cc:288] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [CANDIDATE]: Term 1 election: Requested vote from peers 7d55abc064024922946718f3426ff476 (127.6.31.130:34129), c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865)
I20260927 10:50:14.825518 14092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ecebd5affa4684bb9d58a5b9706eeb" candidate_uuid: "f37bba543be6487ca802300c46bccaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d55abc064024922946718f3426ff476"
I20260927 10:50:14.825649 14232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ecebd5affa4684bb9d58a5b9706eeb" candidate_uuid: "f37bba543be6487ca802300c46bccaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69"
I20260927 10:50:14.825662 14092 raft_consensus.cc:3037] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:14.825802 14232 raft_consensus.cc:3037] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:14.826725 14092 raft_consensus.cc:2454] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f37bba543be6487ca802300c46bccaa7 in term 1.
I20260927 10:50:14.826960 14232 raft_consensus.cc:2454] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f37bba543be6487ca802300c46bccaa7 in term 1.
I20260927 10:50:14.827023 13883 leader_election.cc:302] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, f37bba543be6487ca802300c46bccaa7; no voters:
I20260927 10:50:14.827232 14313 raft_consensus.cc:2781] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:14.827464 14313 raft_consensus.cc:686] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 1 LEADER]: Becoming Leader. State: Replica: f37bba543be6487ca802300c46bccaa7, State: Running, Role: LEADER
I20260927 10:50:14.827770 14313 consensus_queue.cc:237] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.830963 13794 catalog_manager.cc:5495] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f37bba543be6487ca802300c46bccaa7 (127.6.31.129). New cstate: current_term: 1 leader_uuid: "f37bba543be6487ca802300c46bccaa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } health_report { overall_health: UNKNOWN } } }
I20260927 10:50:14.860324 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:14.856888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.6.31.129:45493)
0927 10:50:14.857023 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:14.857028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:14.857043 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:50:14.857156 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:14.857159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:14.857231 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:14.857344 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:14.857578 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.858081 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.858911 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:14.859164 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:14.859271 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:14.859539 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:14.859945 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:14.860067 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:14.860131 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 10:50:14.860183 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:50:14.865834 14092 raft_consensus.cc:1261] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Refusing update from remote peer f37bba543be6487ca802300c46bccaa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:14.866266 14232 raft_consensus.cc:1261] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Refusing update from remote peer f37bba543be6487ca802300c46bccaa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:14.866561 14317 consensus_queue.cc:1035] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:14.866801 14313 consensus_queue.cc:1035] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:14.875407 14320 mvcc.cc:205] Tried to move back new op lower bound from 7333913456083738624 to 7333913455936970752. Current Snapshot: MvccSnapshot[applied={T|T < 7333913456083738624 or (T in {7333913456083738624})}]
I20260927 10:50:14.877331 14321 mvcc.cc:205] Tried to move back new op lower bound from 7333913456083738624 to 7333913455936970752. Current Snapshot: MvccSnapshot[applied={T|T < 7333913456083738624}]
I20260927 10:50:14.881233 13794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:14.881410 13794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:14.883781 13794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e: Sending DeleteTablet for 3 replicas of tablet 07ecebd5affa4684bb9d58a5b9706eeb
I20260927 10:50:14.884619 13932 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ecebd5affa4684bb9d58a5b9706eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:14 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:56492
I20260927 10:50:14.884624 14212 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ecebd5affa4684bb9d58a5b9706eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:14 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:50958
I20260927 10:50:14.884712 14072 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ecebd5affa4684bb9d58a5b9706eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:14 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:52092
I20260927 10:50:14.885422 14330 tablet_replica.cc:330] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: stopping tablet replica
I20260927 10:50:14.885757 14330 raft_consensus.cc:2229] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:50:14.885704 13794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:50:14.886083 13794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:14.886088 14330 raft_consensus.cc:2258] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:14.886471 14331 tablet_replica.cc:330] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: stopping tablet replica
I20260927 10:50:14.886826 14331 raft_consensus.cc:2229] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:14.887209 14331 raft_consensus.cc:2258] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:14.887651 14330 ts_tablet_manager.cc:1905] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:14.888965 14331 ts_tablet_manager.cc:1905] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:14.890450 14330 ts_tablet_manager.cc:1918] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:14.890609 14330 log.cc:1196] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/07ecebd5affa4684bb9d58a5b9706eeb
I20260927 10:50:14.890650 13932 tablet_service.cc:1467] Processing CreateTablet for tablet 8d927072bf584eed97a003de8fe72a70 (DEFAULT_TABLE table=test-table [id=d0e3eed529764929a770670a37a9d172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:14.890890 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d927072bf584eed97a003de8fe72a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:14.890964 14072 tablet_service.cc:1467] Processing CreateTablet for tablet 8d927072bf584eed97a003de8fe72a70 (DEFAULT_TABLE table=test-table [id=d0e3eed529764929a770670a37a9d172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:14.890966 14212 tablet_service.cc:1467] Processing CreateTablet for tablet 8d927072bf584eed97a003de8fe72a70 (DEFAULT_TABLE table=test-table [id=d0e3eed529764929a770670a37a9d172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:14.891216 14072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d927072bf584eed97a003de8fe72a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:14.891233 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d927072bf584eed97a003de8fe72a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:14.893270 14308 tablet_bootstrap.cc:492] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: Bootstrap starting.
I20260927 10:50:14.893509 14307 tablet_bootstrap.cc:492] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: Bootstrap starting.
I20260927 10:50:14.893945 14330 ts_tablet_manager.cc:1939] T 07ecebd5affa4684bb9d58a5b9706eeb P f37bba543be6487ca802300c46bccaa7: Deleting consensus metadata
I20260927 10:50:14.894170 14308 tablet_bootstrap.cc:654] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:14.894488 14307 tablet_bootstrap.cc:654] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:14.895069 14306 tablet_bootstrap.cc:492] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: Bootstrap starting.
I20260927 10:50:14.895782 14308 tablet_bootstrap.cc:492] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: No bootstrap required, opened a new log
I20260927 10:50:14.895749 13779 catalog_manager.cc:4841] TS f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493): tablet 07ecebd5affa4684bb9d58a5b9706eeb (table test-table [id=2a95123a7fa347a6a0410f854873ca26]) successfully deleted
I20260927 10:50:14.895861 14308 ts_tablet_manager.cc:1397] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 10:50:14.896114 14306 tablet_bootstrap.cc:654] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:14.896404 14308 raft_consensus.cc:348] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.896512 14308 raft_consensus.cc:374] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:14.896549 14308 raft_consensus.cc:729] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37bba543be6487ca802300c46bccaa7, State: Initialized, Role: FOLLOWER
I20260927 10:50:14.896688 14308 consensus_queue.cc:260] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.896970 14308 ts_tablet_manager.cc:1428] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:14.897518 14306 tablet_bootstrap.cc:492] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: No bootstrap required, opened a new log
I20260927 10:50:14.897598 14306 ts_tablet_manager.cc:1397] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 10:50:14.898075 14306 raft_consensus.cc:348] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.898186 14306 raft_consensus.cc:374] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:14.898222 14306 raft_consensus.cc:729] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ddb9d52acd4afb8c1d75e3be65cf69, State: Initialized, Role: FOLLOWER
I20260927 10:50:14.898376 14306 consensus_queue.cc:260] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.898634 14306 ts_tablet_manager.cc:1428] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:50:14.898720 14331 ts_tablet_manager.cc:1918] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:14.898816 14331 log.cc:1196] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/07ecebd5affa4684bb9d58a5b9706eeb
I20260927 10:50:14.899137 14331 ts_tablet_manager.cc:1939] T 07ecebd5affa4684bb9d58a5b9706eeb P 7d55abc064024922946718f3426ff476: Deleting consensus metadata
I20260927 10:50:14.899200 14307 tablet_bootstrap.cc:492] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: No bootstrap required, opened a new log
I20260927 10:50:14.899281 14307 ts_tablet_manager.cc:1397] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260927 10:50:14.899747 14307 raft_consensus.cc:348] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.899853 14307 raft_consensus.cc:374] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:14.901319 13779 catalog_manager.cc:4841] TS 7d55abc064024922946718f3426ff476 (127.6.31.130:34129): tablet 07ecebd5affa4684bb9d58a5b9706eeb (table test-table [id=2a95123a7fa347a6a0410f854873ca26]) successfully deleted
I20260927 10:50:14.899885 14307 raft_consensus.cc:729] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d55abc064024922946718f3426ff476, State: Initialized, Role: FOLLOWER
I20260927 10:50:14.902110 14307 consensus_queue.cc:260] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:14.902374 14307 ts_tablet_manager.cc:1428] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:50:14.902745 14329 tablet_replica.cc:330] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: stopping tablet replica
I20260927 10:50:14.903134 14329 raft_consensus.cc:2229] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:14.903524 14329 raft_consensus.cc:2258] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:14.905864 14329 ts_tablet_manager.cc:1905] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:14.908823 14329 ts_tablet_manager.cc:1918] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:14.908928 14329 log.cc:1196] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/07ecebd5affa4684bb9d58a5b9706eeb
I20260927 10:50:14.909224 14329 ts_tablet_manager.cc:1939] T 07ecebd5affa4684bb9d58a5b9706eeb P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting consensus metadata
I20260927 10:50:14.910564 13781 catalog_manager.cc:4841] TS c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865): tablet 07ecebd5affa4684bb9d58a5b9706eeb (table test-table [id=2a95123a7fa347a6a0410f854873ca26]) successfully deleted
W20260927 10:50:14.946128 14279 tablet.cc:2369] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:50:14.983266 14139 tablet.cc:2369] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:50:15.012082 13999 tablet.cc:2369] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:50:15.162369 14314 raft_consensus.cc:482] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:15.162536 14314 raft_consensus.cc:504] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.163743 14314 leader_election.cc:288] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493), c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865)
I20260927 10:50:15.168525 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.164155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:38937 (local address 127.6.31.131:41865)
0927 10:50:15.164290 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:15.164295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:15.164468 (+ 173us) server_negotiation.cc:407] Connection header received
0927 10:50:15.164634 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:15.164637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:15.164694 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:15.164807 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:15.165169 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.165702 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.167542 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.167943 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.168044 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.168085 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:15.168337 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 10:50:15.168410 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:50:15.168552 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.164028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:41865 (local address 127.6.31.130:38937)
0927 10:50:15.164353 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:15.164368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:15.164492 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:15.164823 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:15.164826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:15.164842 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:15.165049 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.165057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.165811 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.165817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.167432 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.167441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.168070 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.168073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.168154 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.168176 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:50:15.168306 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 10:50:15.168384 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":219,"spinlock_wait_cycles":5504,"thread_start_us":94,"threads_started":1}
I20260927 10:50:15.169095 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.164032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:45493 (local address 127.6.31.130:34759)
0927 10:50:15.164410 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:15.164443 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:15.164556 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:15.164833 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:15.164836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:15.164851 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:15.165057 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.165064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.165859 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.165862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.167431 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.167441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.168623 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.168627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.168732 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.168760 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:50:15.168859 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:50:15.168926 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":201,"thread_start_us":69,"threads_started":1}
I20260927 10:50:15.169117 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.164170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:34759 (local address 127.6.31.129:45493)
0927 10:50:15.164393 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:15.164400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:15.164536 (+ 136us) server_negotiation.cc:407] Connection header received
0927 10:50:15.164685 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:15.164687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:15.164726 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:15.164823 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:15.165183 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.165751 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.168131 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.168506 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.168632 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.168705 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:15.168912 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 10:50:15.168990 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260927 10:50:15.169171 14232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d927072bf584eed97a003de8fe72a70" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" is_pre_election: true
I20260927 10:50:15.169332 14232 raft_consensus.cc:2454] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 0.
I20260927 10:50:15.169597 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d927072bf584eed97a003de8fe72a70" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7" is_pre_election: true
I20260927 10:50:15.169677 14025 leader_election.cc:302] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, c4ddb9d52acd4afb8c1d75e3be65cf69; no voters:
I20260927 10:50:15.169831 13952 raft_consensus.cc:2454] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 0.
I20260927 10:50:15.170006 14314 raft_consensus.cc:2781] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:15.170074 14314 raft_consensus.cc:482] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:15.170121 14314 raft_consensus.cc:3037] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:15.171130 14314 raft_consensus.cc:504] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.171515 14314 leader_election.cc:288] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 election: Requested vote from peers f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493), c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865)
I20260927 10:50:15.171838 14232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d927072bf584eed97a003de8fe72a70" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69"
I20260927 10:50:15.171883 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d927072bf584eed97a003de8fe72a70" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7"
I20260927 10:50:15.171975 14232 raft_consensus.cc:3037] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:15.172008 13952 raft_consensus.cc:3037] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:15.172895 13952 raft_consensus.cc:2454] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 1.
I20260927 10:50:15.172907 14232 raft_consensus.cc:2454] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 1.
I20260927 10:50:15.173202 14023 leader_election.cc:302] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, f37bba543be6487ca802300c46bccaa7; no voters:
I20260927 10:50:15.173374 14314 raft_consensus.cc:2781] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:15.173560 14314 raft_consensus.cc:686] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 1 LEADER]: Becoming Leader. State: Replica: 7d55abc064024922946718f3426ff476, State: Running, Role: LEADER
I20260927 10:50:15.173825 14314 consensus_queue.cc:237] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.176225 13795 catalog_manager.cc:5495] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d55abc064024922946718f3426ff476 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:50:15 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506213, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.190@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:15.213687 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.204932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.6.31.190:43083)
0927 10:50:15.205133 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:15.205136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:15.205283 (+ 147us) server_negotiation.cc:407] Connection header received
0927 10:50:15.205398 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:15.205401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:15.205451 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:15.205565 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:15.206218 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.206733 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.207523 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.207728 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.207806 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.210159 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:15.210162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:15.210164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:15.210182 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:15.211900 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:15.212407 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:15.212412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:15.212414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:15.212491 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:15.212764 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:15.212770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:15.212772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:15.212946 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:15.213051 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:15.213344 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 10:50:15.213510 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":56}
I20260927 10:50:15.213688 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.204891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43083 (local address 127.0.0.1:54362)
0927 10:50:15.205168 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:15.205206 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:15.205306 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:15.205586 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:15.205588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:15.205972 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:15.206069 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.206075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.206830 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.206832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.207428 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.207435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.207826 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.207828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.207885 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.208496 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:15.208508 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:15.210011 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:15.212038 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:15.212049 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:15.212057 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:15.212296 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:15.212595 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:15.212598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:15.212600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:15.212678 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:15.213051 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:15.213060 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:15.213176 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 10:50:15.213291 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 10:50:15.213456 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":110,"threads_started":1}
I20260927 10:50:15.217901 13795 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:54362:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:50:15.218536 13795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:15.223194 14072 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3c5f269ba140bcbc4ddc40e59aca9c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2104ff07f275446088ec72e53f02b603]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:50:15.223184 13932 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3c5f269ba140bcbc4ddc40e59aca9c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2104ff07f275446088ec72e53f02b603]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:50:15.223259 14212 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3c5f269ba140bcbc4ddc40e59aca9c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2104ff07f275446088ec72e53f02b603]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:50:15.223577 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3c5f269ba140bcbc4ddc40e59aca9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:15.223578 14072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3c5f269ba140bcbc4ddc40e59aca9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:15.223578 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3c5f269ba140bcbc4ddc40e59aca9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:15.226061 14307 tablet_bootstrap.cc:492] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: Bootstrap starting.
I20260927 10:50:15.226239 14306 tablet_bootstrap.cc:492] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: Bootstrap starting.
I20260927 10:50:15.227381 14306 tablet_bootstrap.cc:654] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:15.228245 14307 tablet_bootstrap.cc:654] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:15.229049 14308 tablet_bootstrap.cc:492] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: Bootstrap starting.
I20260927 10:50:15.229436 14307 tablet_bootstrap.cc:492] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: No bootstrap required, opened a new log
I20260927 10:50:15.229528 14307 ts_tablet_manager.cc:1397] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 10:50:15.229768 14306 tablet_bootstrap.cc:492] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: No bootstrap required, opened a new log
I20260927 10:50:15.229859 14306 ts_tablet_manager.cc:1397] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 10:50:15.229954 14308 tablet_bootstrap.cc:654] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:15.230022 14307 raft_consensus.cc:348] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.230160 14307 raft_consensus.cc:374] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:15.230201 14307 raft_consensus.cc:729] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d55abc064024922946718f3426ff476, State: Initialized, Role: FOLLOWER
I20260927 10:50:15.230321 14307 consensus_queue.cc:260] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.230355 14306 raft_consensus.cc:348] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.230466 14306 raft_consensus.cc:374] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:15.230504 14306 raft_consensus.cc:729] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ddb9d52acd4afb8c1d75e3be65cf69, State: Initialized, Role: FOLLOWER
I20260927 10:50:15.230571 14314 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.230695 14314 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:50:15.230629 14306 consensus_queue.cc:260] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.230911 14307 ts_tablet_manager.cc:1428] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 10:50:15.231009 14308 tablet_bootstrap.cc:492] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: No bootstrap required, opened a new log
I20260927 10:50:15.231143 14308 ts_tablet_manager.cc:1397] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:50:15.231163 14312 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.231287 14312 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:50:15.231551 14308 raft_consensus.cc:348] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.231674 14308 raft_consensus.cc:374] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:15.231709 14308 raft_consensus.cc:729] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37bba543be6487ca802300c46bccaa7, State: Initialized, Role: FOLLOWER
I20260927 10:50:15.231839 14308 consensus_queue.cc:260] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.232121 14308 ts_tablet_manager.cc:1428] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:15.232066 14313 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.232183 14313 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:50:15.232928 14306 ts_tablet_manager.cc:1428] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:50:15.457206 14314 raft_consensus.cc:482] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:15.457361 14314 raft_consensus.cc:504] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.457875 14314 leader_election.cc:288] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493), c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865)
I20260927 10:50:15.458365 14232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c5f269ba140bcbc4ddc40e59aca9c" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" is_pre_election: true
I20260927 10:50:15.458369 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c5f269ba140bcbc4ddc40e59aca9c" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7" is_pre_election: true
I20260927 10:50:15.458547 14232 raft_consensus.cc:2454] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 0.
I20260927 10:50:15.458559 13952 raft_consensus.cc:2454] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 0.
I20260927 10:50:15.458882 14023 leader_election.cc:302] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, f37bba543be6487ca802300c46bccaa7; no voters:
I20260927 10:50:15.459105 14314 raft_consensus.cc:2781] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:15.459172 14314 raft_consensus.cc:482] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:15.459218 14314 raft_consensus.cc:3037] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:15.460227 14314 raft_consensus.cc:504] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.460575 14314 leader_election.cc:288] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 election: Requested vote from peers f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493), c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865)
I20260927 10:50:15.460863 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c5f269ba140bcbc4ddc40e59aca9c" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7"
I20260927 10:50:15.460927 14232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c5f269ba140bcbc4ddc40e59aca9c" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69"
I20260927 10:50:15.460976 13952 raft_consensus.cc:3037] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:15.461062 14232 raft_consensus.cc:3037] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:15.461897 14232 raft_consensus.cc:2454] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 1.
I20260927 10:50:15.461906 13952 raft_consensus.cc:2454] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 1.
I20260927 10:50:15.462172 14025 leader_election.cc:302] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, c4ddb9d52acd4afb8c1d75e3be65cf69; no voters:
I20260927 10:50:15.462409 14314 raft_consensus.cc:2781] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:15.462497 14314 raft_consensus.cc:686] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [term 1 LEADER]: Becoming Leader. State: Replica: 7d55abc064024922946718f3426ff476, State: Running, Role: LEADER
I20260927 10:50:15.462631 14314 consensus_queue.cc:237] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.463304 14337 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: TxnStatusTablet state changed. Reason: New leader 7d55abc064024922946718f3426ff476. Latest consensus state: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.463420 14337 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:50:15.463632 14349 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:50:15.463785 14349 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:50:15.464288 13795 catalog_manager.cc:5495] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d55abc064024922946718f3426ff476 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } health_report { overall_health: UNKNOWN } } }
I20260927 10:50:15.525269 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.521492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34129 (local address 127.0.0.1:52096)
0927 10:50:15.521618 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:15.521635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:15.521767 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:15.522144 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:15.522147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:15.522157 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:50:15.522273 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.522278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.523014 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.523019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.523689 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.523698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.524156 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.524160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.524249 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.524299 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:50:15.524973 (+ 674us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:50:15.524987 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 10:50:15.525052 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 10:50:15.525137 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 10:50:15.525283 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.521625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.6.31.130:34129)
0927 10:50:15.521881 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:15.521885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:15.521906 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:50:15.521964 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:15.521966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:15.522020 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:15.522134 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:15.522379 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.522907 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.523810 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.524048 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.524158 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.524467 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:15.524878 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:15.524999 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:15.525061 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 10:50:15.525110 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260927 10:50:15.528448 14232 raft_consensus.cc:1261] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Refusing update from remote peer 7d55abc064024922946718f3426ff476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:15.528648 13952 raft_consensus.cc:1261] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Refusing update from remote peer 7d55abc064024922946718f3426ff476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:15.529047 14314 consensus_queue.cc:1035] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:15.529201 14337 consensus_queue.cc:1035] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:15.530898 14313 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: TxnStatusTablet state changed. Reason: New leader 7d55abc064024922946718f3426ff476. Latest consensus state: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.531018 14313 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:50:15.531188 14312 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: TxnStatusTablet state changed. Reason: New leader 7d55abc064024922946718f3426ff476. Latest consensus state: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.531294 14312 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:50:15.533970 14314 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.534083 14314 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:50:15.534049 14312 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.534152 14312 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P c4ddb9d52acd4afb8c1d75e3be65cf69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:50:15.534142 14313 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.534238 14313 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P f37bba543be6487ca802300c46bccaa7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:50:15.534409 14353 mvcc.cc:205] Tried to move back new op lower bound from 7333913458799398912 to 7333913458536095744. Current Snapshot: MvccSnapshot[applied={T|T < 7333913458799398912}]
I20260927 10:50:15.535120 14314 tablet_replica.cc:439] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } }
I20260927 10:50:15.535224 14314 tablet_replica.cc:442] T 0b3c5f269ba140bcbc4ddc40e59aca9c P 7d55abc064024922946718f3426ff476: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:50:15.543340 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.540160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.6.31.130:34129)
0927 10:50:15.540295 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:15.540300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:15.540416 (+ 116us) server_negotiation.cc:407] Connection header received
0927 10:50:15.540539 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:15.540543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:15.540586 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:15.540662 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:15.540925 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.541450 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.542295 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.542528 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.542650 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.542899 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:15.542973 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:15.543052 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:15.543165 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 10:50:15.543212 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:50:15.554181 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.550175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:48999 (local address 127.6.31.130:34129)
0927 10:50:15.550284 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:15.550287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:15.550457 (+ 170us) server_negotiation.cc:407] Connection header received
0927 10:50:15.550571 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:15.550573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:15.550613 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:15.550677 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:15.550996 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.551572 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.552810 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.553115 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.553213 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.553616 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:15.553689 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:15.553756 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:15.553957 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 10:50:15.554012 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":41}
I20260927 10:50:15.554201 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.550049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34129 (local address 127.6.31.130:48999)
0927 10:50:15.550371 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:15.550385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:15.550484 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:15.550717 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:15.550721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:15.550745 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:50:15.550896 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.550902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.551676 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.551680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.552652 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:15.552662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.553236 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.553239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:15.553373 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.553423 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:50:15.553808 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:50:15.553818 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 10:50:15.553944 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 10:50:15.554020 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":90,"threads_started":1}
I20260927 10:50:15.560902 14232 raft_consensus.cc:1261] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Refusing update from remote peer 7d55abc064024922946718f3426ff476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:15.560928 13952 raft_consensus.cc:1261] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Refusing update from remote peer 7d55abc064024922946718f3426ff476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:15.561287 14358 consensus_queue.cc:1035] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:15.561492 14314 consensus_queue.cc:1035] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:15.600467 13794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:15.600646 13794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:15.602954 13794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e: Sending DeleteTablet for 3 replicas of tablet 8d927072bf584eed97a003de8fe72a70
I20260927 10:50:15.603726 14212 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d927072bf584eed97a003de8fe72a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:15 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:50958
I20260927 10:50:15.603740 13932 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d927072bf584eed97a003de8fe72a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:15 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:56492
I20260927 10:50:15.604139 14371 tablet_replica.cc:330] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: stopping tablet replica
I20260927 10:50:15.604349 14371 raft_consensus.cc:2229] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:15.604489 14371 raft_consensus.cc:2258] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:15.605106 14370 tablet_replica.cc:330] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: stopping tablet replica
I20260927 10:50:15.605324 14370 raft_consensus.cc:2229] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:15.605472 14370 raft_consensus.cc:2258] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:15.606324 14370 ts_tablet_manager.cc:1905] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:15.607256 14072 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d927072bf584eed97a003de8fe72a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:15 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:52092
I20260927 10:50:15.607622 14373 tablet_replica.cc:330] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: stopping tablet replica
I20260927 10:50:15.607802 14373 raft_consensus.cc:2229] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:50:15.608052 14371 ts_tablet_manager.cc:1905] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:15.608192 14373 raft_consensus.cc:2258] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:15.608886 14373 ts_tablet_manager.cc:1905] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:15.611367 14371 ts_tablet_manager.cc:1918] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:50:15.611515 14371 log.cc:1196] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/8d927072bf584eed97a003de8fe72a70
I20260927 10:50:15.611900 14371 ts_tablet_manager.cc:1939] T 8d927072bf584eed97a003de8fe72a70 P f37bba543be6487ca802300c46bccaa7: Deleting consensus metadata
I20260927 10:50:15.612022 14373 ts_tablet_manager.cc:1918] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:50:15.612159 14373 log.cc:1196] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/8d927072bf584eed97a003de8fe72a70
I20260927 10:50:15.612488 14373 ts_tablet_manager.cc:1939] T 8d927072bf584eed97a003de8fe72a70 P 7d55abc064024922946718f3426ff476: Deleting consensus metadata
I20260927 10:50:15.612854 13779 catalog_manager.cc:4841] TS f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493): tablet 8d927072bf584eed97a003de8fe72a70 (table test-table [id=d0e3eed529764929a770670a37a9d172]) successfully deleted
I20260927 10:50:15.613224 13779 catalog_manager.cc:4841] TS 7d55abc064024922946718f3426ff476 (127.6.31.130:34129): tablet 8d927072bf584eed97a003de8fe72a70 (table test-table [id=d0e3eed529764929a770670a37a9d172]) successfully deleted
I20260927 10:50:15.615557 14370 ts_tablet_manager.cc:1918] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:50:15.615657 14370 log.cc:1196] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/8d927072bf584eed97a003de8fe72a70
I20260927 10:50:15.615906 14370 ts_tablet_manager.cc:1939] T 8d927072bf584eed97a003de8fe72a70 P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting consensus metadata
I20260927 10:50:15.616622 13781 catalog_manager.cc:4841] TS c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865): tablet 8d927072bf584eed97a003de8fe72a70 (table test-table [id=d0e3eed529764929a770670a37a9d172]) successfully deleted
Sep 27 10:50:15 dist-test-slave-c60x krb5kdc[13750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:50:15 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:15 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:50:15.633126 13844 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:15.626445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.6.31.190:43083)
0927 10:50:15.626603 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:15.626608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:15.626819 (+ 211us) server_negotiation.cc:407] Connection header received
0927 10:50:15.627002 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:15.627005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:15.627050 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:15.627179 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:15.628097 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.628680 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.629552 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.629775 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.629884 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.632991 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54366: BlockingRecv error: recv got EOF from 127.0.0.1:54366 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:50:15 dist-test-slave-c60x krb5kdc[13750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM
I20260927 10:50:15.643626 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.634618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.6.31.190:43083)
0927 10:50:15.634726 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:15.634730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:15.634853 (+ 123us) server_negotiation.cc:407] Connection header received
0927 10:50:15.635001 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:15.635004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:15.635051 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:15.635161 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:15.635845 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.636647 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.637470 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.637819 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.637909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.640323 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:15.640327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:15.640329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:15.640356 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:15.641917 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:15.642363 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:15.642366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:15.642367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:15.642454 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:15.642720 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:15.642723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:15.642725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:15.642935 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:15.643064 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:15.643360 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 10:50:15.643469 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:50:15.646144 13794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:50:15.646597 13794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:15.651479 13932 tablet_service.cc:1467] Processing CreateTablet for tablet 1989c71fa1f2447eba55d8a7f0a2faff (DEFAULT_TABLE table=test-table [id=70e28a7567664850b570f3129381d9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:15.651628 14072 tablet_service.cc:1467] Processing CreateTablet for tablet 1989c71fa1f2447eba55d8a7f0a2faff (DEFAULT_TABLE table=test-table [id=70e28a7567664850b570f3129381d9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:15.651803 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1989c71fa1f2447eba55d8a7f0a2faff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:15.651878 14072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1989c71fa1f2447eba55d8a7f0a2faff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:15.653702 14307 tablet_bootstrap.cc:492] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: Bootstrap starting.
I20260927 10:50:15.654188 14308 tablet_bootstrap.cc:492] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: Bootstrap starting.
I20260927 10:50:15.654362 14212 tablet_service.cc:1467] Processing CreateTablet for tablet 1989c71fa1f2447eba55d8a7f0a2faff (DEFAULT_TABLE table=test-table [id=70e28a7567664850b570f3129381d9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:15.654568 14307 tablet_bootstrap.cc:654] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:15.654637 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1989c71fa1f2447eba55d8a7f0a2faff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:15.655121 14308 tablet_bootstrap.cc:654] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:15.656232 14307 tablet_bootstrap.cc:492] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: No bootstrap required, opened a new log
I20260927 10:50:15.656337 14307 ts_tablet_manager.cc:1397] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:50:15.656697 14307 raft_consensus.cc:348] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.656814 14307 raft_consensus.cc:374] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:15.656855 14307 raft_consensus.cc:729] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d55abc064024922946718f3426ff476, State: Initialized, Role: FOLLOWER
I20260927 10:50:15.656949 14306 tablet_bootstrap.cc:492] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: Bootstrap starting.
I20260927 10:50:15.656975 14307 consensus_queue.cc:260] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.657243 14307 ts_tablet_manager.cc:1428] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:15.657823 14306 tablet_bootstrap.cc:654] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:15.659013 14306 tablet_bootstrap.cc:492] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: No bootstrap required, opened a new log
I20260927 10:50:15.659166 14306 ts_tablet_manager.cc:1397] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:50:15.659577 14308 tablet_bootstrap.cc:492] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: No bootstrap required, opened a new log
I20260927 10:50:15.659657 14308 ts_tablet_manager.cc:1397] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 10:50:15.659633 14306 raft_consensus.cc:348] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.659758 14306 raft_consensus.cc:374] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:15.659790 14306 raft_consensus.cc:729] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ddb9d52acd4afb8c1d75e3be65cf69, State: Initialized, Role: FOLLOWER
I20260927 10:50:15.659921 14306 consensus_queue.cc:260] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.660022 14308 raft_consensus.cc:348] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.660125 14308 raft_consensus.cc:374] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:15.660156 14308 raft_consensus.cc:729] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37bba543be6487ca802300c46bccaa7, State: Initialized, Role: FOLLOWER
I20260927 10:50:15.660198 14306 ts_tablet_manager.cc:1428] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:15.660265 14308 consensus_queue.cc:260] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.660498 14308 ts_tablet_manager.cc:1428] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:50:15.887723 14314 raft_consensus.cc:482] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:15.887867 14314 raft_consensus.cc:504] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.888257 14314 leader_election.cc:288] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493), c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865)
I20260927 10:50:15.888753 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1989c71fa1f2447eba55d8a7f0a2faff" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" is_pre_election: true
I20260927 10:50:15.888782 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1989c71fa1f2447eba55d8a7f0a2faff" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7" is_pre_election: true
I20260927 10:50:15.888913 14231 raft_consensus.cc:2454] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 0.
I20260927 10:50:15.888949 13952 raft_consensus.cc:2454] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 0.
I20260927 10:50:15.889276 14023 leader_election.cc:302] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, f37bba543be6487ca802300c46bccaa7; no voters:
I20260927 10:50:15.889497 14314 raft_consensus.cc:2781] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:15.889562 14314 raft_consensus.cc:482] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:15.889590 14314 raft_consensus.cc:3037] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:15.890640 14314 raft_consensus.cc:504] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.890990 14314 leader_election.cc:288] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 election: Requested vote from peers f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493), c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865)
I20260927 10:50:15.891351 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1989c71fa1f2447eba55d8a7f0a2faff" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7"
I20260927 10:50:15.891475 13952 raft_consensus.cc:3037] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:15.892614 13952 raft_consensus.cc:2454] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 1.
I20260927 10:50:15.892859 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1989c71fa1f2447eba55d8a7f0a2faff" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69"
I20260927 10:50:15.892910 14023 leader_election.cc:302] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, f37bba543be6487ca802300c46bccaa7; no voters:
I20260927 10:50:15.893016 14231 raft_consensus.cc:3037] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:15.893090 14314 raft_consensus.cc:2781] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:15.893208 14314 raft_consensus.cc:686] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 1 LEADER]: Becoming Leader. State: Replica: 7d55abc064024922946718f3426ff476, State: Running, Role: LEADER
I20260927 10:50:15.893370 14314 consensus_queue.cc:237] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } }
I20260927 10:50:15.894097 14231 raft_consensus.cc:2454] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 1.
I20260927 10:50:15.895151 13794 catalog_manager.cc:5495] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d55abc064024922946718f3426ff476 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "7d55abc064024922946718f3426ff476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } health_report { overall_health: UNKNOWN } } }
I20260927 10:50:15.958572 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:15.955552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.6.31.130:34129)
0927 10:50:15.955676 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:15.955680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:15.955696 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:50:15.955826 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:15.955828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:15.955873 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:15.955936 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:15.956173 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.956690 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.957563 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:15.957810 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:15.957890 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:15.958199 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:15.958264 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:15.958317 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:15.958431 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 10:50:15.958469 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:50:15.960561 13952 raft_consensus.cc:1261] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Refusing update from remote peer 7d55abc064024922946718f3426ff476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:15.960557 14231 raft_consensus.cc:1261] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Refusing update from remote peer 7d55abc064024922946718f3426ff476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:15.961187 14358 consensus_queue.cc:1035] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:15.961337 14314 consensus_queue.cc:1035] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:15.968144 13792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:15.968286 13792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:15.970950 13792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e: Sending DeleteTablet for 3 replicas of tablet 1989c71fa1f2447eba55d8a7f0a2faff
I20260927 10:50:15.971663 13932 tablet_service.cc:1514] Processing DeleteTablet for tablet 1989c71fa1f2447eba55d8a7f0a2faff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:15 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:56492
I20260927 10:50:15.971675 14212 tablet_service.cc:1514] Processing DeleteTablet for tablet 1989c71fa1f2447eba55d8a7f0a2faff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:15 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:50958
I20260927 10:50:15.971733 14072 tablet_service.cc:1514] Processing DeleteTablet for tablet 1989c71fa1f2447eba55d8a7f0a2faff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:15 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:52092
I20260927 10:50:15.971895 14371 tablet_replica.cc:330] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: stopping tablet replica
I20260927 10:50:15.972012 14371 raft_consensus.cc:2229] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:15.972160 14371 raft_consensus.cc:2258] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:15.972344 14370 tablet_replica.cc:330] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: stopping tablet replica
I20260927 10:50:15.972486 14370 raft_consensus.cc:2229] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:15.972476 13792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:50:15.972640 14370 raft_consensus.cc:2258] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:15.972815 14373 tablet_replica.cc:330] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: stopping tablet replica
W20260927 10:50:15.972880 13792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:15.972946 14373 raft_consensus.cc:2229] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:50:15.973192 14373 raft_consensus.cc:2258] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:15.973333 14370 ts_tablet_manager.cc:1905] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:15.975670 14370 ts_tablet_manager.cc:1918] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:15.975740 14370 log.cc:1196] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/1989c71fa1f2447eba55d8a7f0a2faff
I20260927 10:50:15.975972 14370 ts_tablet_manager.cc:1939] T 1989c71fa1f2447eba55d8a7f0a2faff P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting consensus metadata
I20260927 10:50:15.976643 13781 catalog_manager.cc:4841] TS c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865): tablet 1989c71fa1f2447eba55d8a7f0a2faff (table test-table [id=70e28a7567664850b570f3129381d9d7]) successfully deleted
I20260927 10:50:15.977005 14373 ts_tablet_manager.cc:1905] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:15.978052 14212 tablet_service.cc:1467] Processing CreateTablet for tablet 2257629f81c1483ebb146870b17422c1 (DEFAULT_TABLE table=test-table [id=1e9370a93e8c415d82fc5ad1a7b90645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:15.978313 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2257629f81c1483ebb146870b17422c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:15.979218 14373 ts_tablet_manager.cc:1918] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:15.979290 14373 log.cc:1196] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/1989c71fa1f2447eba55d8a7f0a2faff
I20260927 10:50:15.979519 14373 ts_tablet_manager.cc:1939] T 1989c71fa1f2447eba55d8a7f0a2faff P 7d55abc064024922946718f3426ff476: Deleting consensus metadata
I20260927 10:50:15.980262 14306 tablet_bootstrap.cc:492] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: Bootstrap starting.
I20260927 10:50:15.980469 13779 catalog_manager.cc:4841] TS 7d55abc064024922946718f3426ff476 (127.6.31.130:34129): tablet 1989c71fa1f2447eba55d8a7f0a2faff (table test-table [id=70e28a7567664850b570f3129381d9d7]) successfully deleted
I20260927 10:50:15.980973 13932 tablet_service.cc:1467] Processing CreateTablet for tablet 2257629f81c1483ebb146870b17422c1 (DEFAULT_TABLE table=test-table [id=1e9370a93e8c415d82fc5ad1a7b90645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:15.981050 14072 tablet_service.cc:1467] Processing CreateTablet for tablet 2257629f81c1483ebb146870b17422c1 (DEFAULT_TABLE table=test-table [id=1e9370a93e8c415d82fc5ad1a7b90645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:15.981127 14306 tablet_bootstrap.cc:654] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:15.981242 14072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2257629f81c1483ebb146870b17422c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:15.981241 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2257629f81c1483ebb146870b17422c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:15.983371 14307 tablet_bootstrap.cc:492] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: Bootstrap starting.
I20260927 10:50:15.983621 14308 tablet_bootstrap.cc:492] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7: Bootstrap starting.
I20260927 10:50:15.983841 14371 ts_tablet_manager.cc:1905] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:15.984146 14307 tablet_bootstrap.cc:654] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:15.984361 14308 tablet_bootstrap.cc:654] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:15.985208 14307 tablet_bootstrap.cc:492] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: No bootstrap required, opened a new log
I20260927 10:50:15.985291 14307 ts_tablet_manager.cc:1397] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:50:15.985689 14307 raft_consensus.cc:348] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:15.985802 14307 raft_consensus.cc:374] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:15.985834 14307 raft_consensus.cc:729] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d55abc064024922946718f3426ff476, State: Initialized, Role: FOLLOWER
I20260927 10:50:15.985978 14307 consensus_queue.cc:260] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:15.986291 14371 ts_tablet_manager.cc:1918] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:15.986373 14371 log.cc:1196] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/1989c71fa1f2447eba55d8a7f0a2faff
I20260927 10:50:15.986383 14307 ts_tablet_manager.cc:1428] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:15.986760 14371 ts_tablet_manager.cc:1939] T 1989c71fa1f2447eba55d8a7f0a2faff P f37bba543be6487ca802300c46bccaa7: Deleting consensus metadata
I20260927 10:50:15.987282 14306 tablet_bootstrap.cc:492] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: No bootstrap required, opened a new log
I20260927 10:50:15.987423 14306 ts_tablet_manager.cc:1397] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 10:50:15.987919 13779 catalog_manager.cc:4841] TS f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493): tablet 1989c71fa1f2447eba55d8a7f0a2faff (table test-table [id=70e28a7567664850b570f3129381d9d7]) successfully deleted
I20260927 10:50:15.987947 14306 raft_consensus.cc:348] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:15.988130 14306 raft_consensus.cc:374] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:15.988174 14306 raft_consensus.cc:729] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ddb9d52acd4afb8c1d75e3be65cf69, State: Initialized, Role: FOLLOWER
I20260927 10:50:15.988310 14306 consensus_queue.cc:260] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:15.988607 14306 ts_tablet_manager.cc:1428] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:15.989423 14308 tablet_bootstrap.cc:492] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7: No bootstrap required, opened a new log
I20260927 10:50:15.989549 14308 ts_tablet_manager.cc:1397] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 10:50:15.989883 14308 raft_consensus.cc:348] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:15.989985 14308 raft_consensus.cc:374] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:15.990007 14308 raft_consensus.cc:729] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37bba543be6487ca802300c46bccaa7, State: Initialized, Role: FOLLOWER
I20260927 10:50:15.990141 14308 consensus_queue.cc:260] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:15.990433 14308 ts_tablet_manager.cc:1428] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:50:16.305508 14312 raft_consensus.cc:482] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:16.305728 14312 raft_consensus.cc:504] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:16.306921 14312 leader_election.cc:288] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493), 7d55abc064024922946718f3426ff476 (127.6.31.130:34129)
I20260927 10:50:16.310972 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.307307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:58303 (local address 127.6.31.129:45493)
0927 10:50:16.307548 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:16.307552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:16.307594 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:50:16.307782 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:16.307785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:16.307843 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:16.307995 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:16.308281 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.308835 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.309997 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.310408 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.310542 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:16.310603 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:16.310779 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 10:50:16.310854 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260927 10:50:16.310984 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.307232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:45493 (local address 127.6.31.131:58303)
0927 10:50:16.307484 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:16.307502 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:16.307620 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:16.308008 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:16.308012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:16.308033 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:16.308164 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:16.308170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.308956 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.308960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:16.309868 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:16.309877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.310531 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.310535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:16.310641 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:16.310667 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:50:16.310754 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:50:16.310825 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":29,"thread_start_us":101,"threads_started":1}
I20260927 10:50:16.311131 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.307799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:49559 (local address 127.6.31.130:34129)
0927 10:50:16.307964 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:16.307970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:16.307984 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:50:16.308117 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:16.308120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:16.308172 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:16.308248 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:16.308516 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.309023 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.310067 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.310537 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.310671 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:16.310719 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:16.310903 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 10:50:16.310972 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:50:16.311156 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.307623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34129 (local address 127.6.31.131:49559)
0927 10:50:16.307901 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:16.307914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:16.307999 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:16.308268 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:16.308271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:16.308287 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:16.308409 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:16.308414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.309141 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.309146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:16.309937 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:16.309945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.310673 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.310677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:16.310784 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:16.310801 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:50:16.310888 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:50:16.310955 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":20,"thread_start_us":59,"threads_started":1}
I20260927 10:50:16.311442 14314 raft_consensus.cc:482] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:16.311563 14314 raft_consensus.cc:504] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:16.311734 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257629f81c1483ebb146870b17422c1" candidate_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7" is_pre_election: true
I20260927 10:50:16.311735 14092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257629f81c1483ebb146870b17422c1" candidate_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d55abc064024922946718f3426ff476" is_pre_election: true
I20260927 10:50:16.311863 13952 raft_consensus.cc:2454] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4ddb9d52acd4afb8c1d75e3be65cf69 in term 0.
I20260927 10:50:16.311887 14092 raft_consensus.cc:2454] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4ddb9d52acd4afb8c1d75e3be65cf69 in term 0.
I20260927 10:50:16.311966 14314 leader_election.cc:288] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865), f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493)
I20260927 10:50:16.312294 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257629f81c1483ebb146870b17422c1" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7" is_pre_election: true
I20260927 10:50:16.312291 14163 leader_election.cc:302] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, c4ddb9d52acd4afb8c1d75e3be65cf69; no voters:
I20260927 10:50:16.312358 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257629f81c1483ebb146870b17422c1" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" is_pre_election: true
I20260927 10:50:16.312415 13952 raft_consensus.cc:2454] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 0.
I20260927 10:50:16.312464 14231 raft_consensus.cc:2454] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d55abc064024922946718f3426ff476 in term 0.
I20260927 10:50:16.312537 14312 raft_consensus.cc:2781] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:16.312587 14312 raft_consensus.cc:482] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:16.312621 14312 raft_consensus.cc:3037] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:16.312654 14023 leader_election.cc:302] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d55abc064024922946718f3426ff476, f37bba543be6487ca802300c46bccaa7; no voters:
I20260927 10:50:16.312928 14314 raft_consensus.cc:2781] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:16.312978 14314 raft_consensus.cc:482] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:16.313004 14314 raft_consensus.cc:3037] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:16.313709 14312 raft_consensus.cc:504] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:16.314064 14312 leader_election.cc:288] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [CANDIDATE]: Term 1 election: Requested vote from peers f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493), 7d55abc064024922946718f3426ff476 (127.6.31.130:34129)
I20260927 10:50:16.314095 14314 raft_consensus.cc:504] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:16.314438 14092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257629f81c1483ebb146870b17422c1" candidate_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d55abc064024922946718f3426ff476"
I20260927 10:50:16.314504 14314 leader_election.cc:288] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 election: Requested vote from peers c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865), f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493)
I20260927 10:50:16.314602 14092 raft_consensus.cc:2379] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c4ddb9d52acd4afb8c1d75e3be65cf69 in current term 1: Already voted for candidate 7d55abc064024922946718f3426ff476 in this term.
I20260927 10:50:16.314668 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257629f81c1483ebb146870b17422c1" candidate_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7"
I20260927 10:50:16.314760 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257629f81c1483ebb146870b17422c1" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69"
I20260927 10:50:16.314775 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257629f81c1483ebb146870b17422c1" candidate_uuid: "7d55abc064024922946718f3426ff476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37bba543be6487ca802300c46bccaa7"
I20260927 10:50:16.314783 13952 raft_consensus.cc:3037] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:16.314914 14231 raft_consensus.cc:2379] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d55abc064024922946718f3426ff476 in current term 1: Already voted for candidate c4ddb9d52acd4afb8c1d75e3be65cf69 in this term.
I20260927 10:50:16.315238 14023 leader_election.cc:302] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d55abc064024922946718f3426ff476; no voters: c4ddb9d52acd4afb8c1d75e3be65cf69, f37bba543be6487ca802300c46bccaa7
I20260927 10:50:16.315404 14314 raft_consensus.cc:2726] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 10:50:16.315882 13952 raft_consensus.cc:2454] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4ddb9d52acd4afb8c1d75e3be65cf69 in term 1.
I20260927 10:50:16.316188 14163 leader_election.cc:302] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4ddb9d52acd4afb8c1d75e3be65cf69, f37bba543be6487ca802300c46bccaa7; no voters: 7d55abc064024922946718f3426ff476
I20260927 10:50:16.316375 14312 raft_consensus.cc:2781] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:16.316604 14312 raft_consensus.cc:686] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 LEADER]: Becoming Leader. State: Replica: c4ddb9d52acd4afb8c1d75e3be65cf69, State: Running, Role: LEADER
I20260927 10:50:16.316867 14312 consensus_queue.cc:237] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } }
I20260927 10:50:16.319052 13792 catalog_manager.cc:5495] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131). New cstate: current_term: 1 leader_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ddb9d52acd4afb8c1d75e3be65cf69" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 41865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 } health_report { overall_health: UNKNOWN } } }
I20260927 10:50:16.362046 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.358531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.6.31.131:41865)
0927 10:50:16.358824 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:16.358831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:16.358843 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:50:16.358913 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:16.358915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:16.358966 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:16.359067 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:16.359344 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.359887 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.360734 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.360970 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.361044 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:16.361263 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:16.361728 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:16.361815 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:16.361876 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 10:50:16.361921 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":133,"threads_started":1}
I20260927 10:50:16.372761 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.368964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34129 (local address 127.6.31.131:51451)
0927 10:50:16.369322 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:16.369341 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:16.369446 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:16.369726 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:16.369729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:16.369737 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:50:16.369858 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:16.369863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.370616 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.370626 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:16.371426 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:16.371435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.371955 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.371958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:16.372034 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:16.372079 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:50:16.372482 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:50:16.372487 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:50:16.372550 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 10:50:16.372617 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":127,"threads_started":1}
I20260927 10:50:16.372766 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.369275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:51451 (local address 127.6.31.130:34129)
0927 10:50:16.369424 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:16.369428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:16.369443 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:50:16.369561 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:16.369564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:16.369617 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:16.369704 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:16.369959 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.370465 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.371543 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.371815 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.371936 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:16.372314 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:16.372393 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:16.372461 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:16.372609 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 10:50:16.372663 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:50:16.379477 13952 raft_consensus.cc:1261] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Refusing update from remote peer c4ddb9d52acd4afb8c1d75e3be65cf69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:16.379498 14092 raft_consensus.cc:1261] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Refusing update from remote peer c4ddb9d52acd4afb8c1d75e3be65cf69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:16.379949 14312 consensus_queue.cc:1035] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d55abc064024922946718f3426ff476" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:16.380400 14398 consensus_queue.cc:1035] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37bba543be6487ca802300c46bccaa7" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:16.397321 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.393452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:41865 (local address 127.6.31.130:42257)
0927 10:50:16.393701 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:16.393716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:16.393841 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:16.394328 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:16.394331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:16.394339 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:50:16.394455 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:16.394461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.395243 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.395246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:16.395868 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:16.395877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.396420 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.396424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:16.396506 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:16.396541 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:50:16.397000 (+ 459us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:50:16.397006 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:50:16.397075 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 10:50:16.397177 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260927 10:50:16.397324 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.393485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:42257 (local address 127.6.31.131:41865)
0927 10:50:16.393601 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:16.393606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:16.393815 (+ 209us) server_negotiation.cc:407] Connection header received
0927 10:50:16.394152 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:16.394158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:16.394216 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:16.394288 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:16.394547 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.395115 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.395972 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:16.396304 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:16.396401 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:16.396757 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:16.396863 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:16.396959 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:16.397108 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 10:50:16.397162 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:50:16.422019 13793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:16.422220 13793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:16.424501 13793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e8013458fef47058ef34dbd1890385e: Sending DeleteTablet for 3 replicas of tablet 2257629f81c1483ebb146870b17422c1
I20260927 10:50:16.425055 14212 tablet_service.cc:1514] Processing DeleteTablet for tablet 2257629f81c1483ebb146870b17422c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:16 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:50958
I20260927 10:50:16.425243 14370 tablet_replica.cc:330] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: stopping tablet replica
I20260927 10:50:16.425367 14370 raft_consensus.cc:2229] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:50:16.425401 13932 tablet_service.cc:1514] Processing DeleteTablet for tablet 2257629f81c1483ebb146870b17422c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:16 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:56492
I20260927 10:50:16.425563 14371 tablet_replica.cc:330] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7: stopping tablet replica
I20260927 10:50:16.425663 14370 raft_consensus.cc:2258] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:16.425665 14371 raft_consensus.cc:2229] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:16.425771 14371 raft_consensus.cc:2258] T 2257629f81c1483ebb146870b17422c1 P f37bba543be6487ca802300c46bccaa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:16.425977 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 13867
I20260927 10:50:16.427042 14072 tablet_service.cc:1514] Processing DeleteTablet for tablet 2257629f81c1483ebb146870b17422c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:16 UTC) from {username='oryx', principal='oryx/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:52092
I20260927 10:50:16.427281 14373 tablet_replica.cc:330] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: stopping tablet replica
I20260927 10:50:16.427420 14373 raft_consensus.cc:2229] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:16.427536 14373 raft_consensus.cc:2258] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:16.428498 14373 ts_tablet_manager.cc:1905] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:16.431164 14373 ts_tablet_manager.cc:1918] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:50:16.431255 14373 log.cc:1196] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/2257629f81c1483ebb146870b17422c1
I20260927 10:50:16.431571 14373 ts_tablet_manager.cc:1939] T 2257629f81c1483ebb146870b17422c1 P 7d55abc064024922946718f3426ff476: Deleting consensus metadata
I20260927 10:50:16.432526 13779 catalog_manager.cc:4841] TS 7d55abc064024922946718f3426ff476 (127.6.31.130:34129): tablet 2257629f81c1483ebb146870b17422c1 (table test-table [id=1e9370a93e8c415d82fc5ad1a7b90645]) successfully deleted
W20260927 10:50:16.434357 13779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:45493 (error 108)
I20260927 10:50:16.434468 14370 ts_tablet_manager.cc:1905] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:16.434725 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 14007
I20260927 10:50:16.436980 14370 ts_tablet_manager.cc:1918] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:50:16.437065 14370 log.cc:1196] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/2257629f81c1483ebb146870b17422c1
I20260927 10:50:16.437263 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:16.435378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:45493 (local address 127.0.0.1:56514)
0927 10:50:16.437030 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:16.437130 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":57,"threads_started":1}
I20260927 10:50:16.437325 14370 ts_tablet_manager.cc:1939] T 2257629f81c1483ebb146870b17422c1 P c4ddb9d52acd4afb8c1d75e3be65cf69: Deleting consensus metadata
W20260927 10:50:16.437446 13779 catalog_manager.cc:4584] TS f37bba543be6487ca802300c46bccaa7 (127.6.31.129:45493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2257629f81c1483ebb146870b17422c1: Network error: Client connection negotiation failed: client connection to 127.6.31.129:45493: connect: Connection refused (error 111)
I20260927 10:50:16.438051 13781 catalog_manager.cc:4841] TS c4ddb9d52acd4afb8c1d75e3be65cf69 (127.6.31.131:41865): tablet 2257629f81c1483ebb146870b17422c1 (table test-table [id=1e9370a93e8c415d82fc5ad1a7b90645]) successfully deleted
I20260927 10:50:16.445715 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 14147
I20260927 10:50:16.452037 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 13763
2026-09-27T10:50:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3849 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:50:16 dist-test-slave-c60x krb5kdc[14415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:50:16 dist-test-slave-c60x krb5kdc[14415](info): set up 2 sockets
Sep 27 10:50:16 dist-test-slave-c60x krb5kdc[14415](info): commencing operation
krb5kdc: starting...
W20260927 10:50:17.523988 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:50:17 dist-test-slave-c60x krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:50:17.654876 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:39577
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:39577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:17.763644 14428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:17.763967 14428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:17.764043 14428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:17.768083 14428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:17.768261 14428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:17.768321 14428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:17.768364 14428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:17.768401 14428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:17.773742 14428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:39577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:39577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:17.774856 14428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:17.775903 14428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:17.784435 14433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:17.784442 14434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:17.784497 14428 server_base.cc:1034] running on GCE node
W20260927 10:50:17.784442 14436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:17.785211 14428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:17.786320 14428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:17.787504 14428 hybrid_clock.cc:648] HybridClock initialized: now 1790506217787497 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T10:50:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1349 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:50:17 dist-test-slave-c60x krb5kdc[14441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:50:17 dist-test-slave-c60x krb5kdc[14441](info): set up 2 sockets
Sep 27 10:50:17 dist-test-slave-c60x krb5kdc[14441](info): commencing operation
krb5kdc: starting...
W20260927 10:50:18.847900 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:50:18 dist-test-slave-c60x krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:50:18.975023 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33611
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:33611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:19.083318 14454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:19.083578 14454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:19.083619 14454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:19.087636 14454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:19.087690 14454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:19.087709 14454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:19.087725 14454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:19.087740 14454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:19.092917 14454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:33611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:19.093930 14454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:19.094913 14454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:19.103770 14460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:19.103786 14459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:19.103786 14462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:19.103849 14454 server_base.cc:1034] running on GCE node
I20260927 10:50:19.104548 14454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:19.105630 14454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:19.106812 14454 hybrid_clock.cc:648] HybridClock initialized: now 1790506219106783 us; error 38 us; skew 500 ppm
Sep 27 10:50:19 dist-test-slave-c60x krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:19.109634 14454 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:50:19.110460 14454 webserver.cc:460] Webserver started at http://127.6.31.190:46555/ using document root <none> and password file <none>
I20260927 10:50:19.110982 14454 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:19.111040 14454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:19.111259 14454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:19.113210 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "7682808124c040b7ab5b1ce01c727fcc"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "8af4061c2c4410e40c61176f93c39f2b"
server_key_iv: "f5b6c53721987ea375f37067b6ce2e5a"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.113816 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "7682808124c040b7ab5b1ce01c727fcc"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "8af4061c2c4410e40c61176f93c39f2b"
server_key_iv: "f5b6c53721987ea375f37067b6ce2e5a"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.117307 14454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:50:19.119550 14469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.120710 14454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:19.120814 14454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "7682808124c040b7ab5b1ce01c727fcc"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "8af4061c2c4410e40c61176f93c39f2b"
server_key_iv: "f5b6c53721987ea375f37067b6ce2e5a"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.120890 14454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:19.134985 14454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:19.137673 14454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:19.137832 14454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:19.144361 14454 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:33611
I20260927 10:50:19.144419 14521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:33611 every 8 connection(s)
I20260927 10:50:19.145293 14454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:19.148138 14522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:19.150954 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 14454
I20260927 10:50:19.151113 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:19.151401 6270 external_mini_cluster.cc:1428] Setting key a0de2c36066e3ace264b3d45b9e9b501
I20260927 10:50:19.154021 14522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: Bootstrap starting.
I20260927 10:50:19.156555 14522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:19.157436 14522 log.cc:824] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:19.159492 14522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: No bootstrap required, opened a new log
Sep 27 10:50:19 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:50:19.162770 14522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } }
I20260927 10:50:19.163014 14522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:19.163107 14522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7682808124c040b7ab5b1ce01c727fcc, State: Initialized, Role: FOLLOWER
I20260927 10:50:19.163542 14522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } }
I20260927 10:50:19.163686 14522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:19.163771 14522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:19.163885 14522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:19.164741 14522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } }
I20260927 10:50:19.165128 14522 leader_election.cc:302] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7682808124c040b7ab5b1ce01c727fcc; no voters:
I20260927 10:50:19.165478 14522 leader_election.cc:288] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:19.165573 14527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:19.165774 14527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 1 LEADER]: Becoming Leader. State: Replica: 7682808124c040b7ab5b1ce01c727fcc, State: Running, Role: LEADER
I20260927 10:50:19.166115 14527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } }
I20260927 10:50:19.166405 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:19.152726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.6.31.190:33611)
0927 10:50:19.153135 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:19.153150 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:19.153184 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:50:19.153786 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:19.153801 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:19.154152 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:19.154432 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:19.155338 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.156337 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.157520 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.158036 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.158212 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:19.160755 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:19.160760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:19.160773 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:19.160800 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:19.163072 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:19.163605 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:19.163611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:19.163621 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:19.163780 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:19.164080 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:19.164086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:19.164089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:19.164467 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:19.164673 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:19.164933 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 10:50:19.165217 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":130,"threads_started":1}
I20260927 10:50:19.166749 14522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:19.167871 14529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7682808124c040b7ab5b1ce01c727fcc. Latest consensus state: current_term: 1 leader_uuid: "7682808124c040b7ab5b1ce01c727fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } } }
I20260927 10:50:19.167999 14529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:19.168843 14536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:19.169186 14528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7682808124c040b7ab5b1ce01c727fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } } }
I20260927 10:50:19.169358 14528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:19.172369 14536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:19.178424 14536 catalog_manager.cc:1347] Generated new cluster ID: ef8c45c0789a4377b82cb187ec4ece00
I20260927 10:50:19.178532 14536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:19.200373 14536 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:19.201695 14536 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:19.209496 14536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: Generated new TSK 0
I20260927 10:50:19.210358 14536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:50:19.251225 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33611
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:19.360733 14548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:19.360981 14548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:19.361065 14548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:19.365195 14548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:19.365253 14548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:19.365358 14548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:19.370848 14548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:19.371979 14548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:19.373019 14548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:50:19.382906 14548 server_base.cc:1034] running on GCE node
W20260927 10:50:19.382851 14556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:19.382858 14554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:19.382854 14553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:19.383530 14548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:19.384092 14548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:19.385257 14548 hybrid_clock.cc:648] HybridClock initialized: now 1790506219385248 us; error 24 us; skew 500 ppm
Sep 27 10:50:19 dist-test-slave-c60x krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:19.388008 14548 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:50:19.388871 14548 webserver.cc:460] Webserver started at http://127.6.31.129:45411/ using document root <none> and password file <none>
I20260927 10:50:19.389439 14548 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:19.389500 14548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:19.389710 14548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:19.391723 14548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "5d8750335a2a4466882ef9777d7413c6"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "a9e9f8e3d7b1a9fb029ba7c09f2b942f"
server_key_iv: "677aedfc01fcf36351d7096b71fa8b94"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.392259 14548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "5d8750335a2a4466882ef9777d7413c6"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "a9e9f8e3d7b1a9fb029ba7c09f2b942f"
server_key_iv: "677aedfc01fcf36351d7096b71fa8b94"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.395855 14548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:19.398319 14563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.399546 14548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:19.399669 14548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "5d8750335a2a4466882ef9777d7413c6"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "a9e9f8e3d7b1a9fb029ba7c09f2b942f"
server_key_iv: "677aedfc01fcf36351d7096b71fa8b94"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.399772 14548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:19.415537 14548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:19.418454 14548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:19.418665 14548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:19.419361 14548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:19.420173 14548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:19.420218 14548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.420284 14548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:19.420315 14548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.429376 14548 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:43569
I20260927 10:50:19.429400 14676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:43569 every 8 connection(s)
I20260927 10:50:19.430215 14548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:19.437472 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 14548
I20260927 10:50:19.437644 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:50:19.437949 6270 external_mini_cluster.cc:1428] Setting key 83c3d2c9fd9b83d128b18deab501be05
Sep 27 10:50:19 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:50:19.444506 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:19.431928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:46661 (local address 127.6.31.190:33611)
0927 10:50:19.432064 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:19.432068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:19.432854 (+ 786us) server_negotiation.cc:407] Connection header received
0927 10:50:19.434167 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:19.434170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:19.434227 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:19.434304 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:19.435457 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.436011 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.437169 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.437434 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.437548 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:19.440376 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:19.440379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:19.440382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:19.440408 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:19.441925 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:19.442540 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:19.442543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:19.442544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:19.442636 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:19.443027 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:19.443030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:19.443031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:19.443503 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:19.443583 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:19.444276 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 10:50:19.444384 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:50:19.445835 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:19.432282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:46661)
0927 10:50:19.432703 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:19.432749 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:19.433952 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:19.434460 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:19.434466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:19.434816 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:19.435305 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:19.435318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.436170 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.436179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:19.437035 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:19.437043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.437586 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.437589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:19.437816 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:19.438484 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:19.438500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:19.440211 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:19.442040 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:19.442049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:19.442066 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:19.442441 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:19.442729 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:19.442735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:19.442740 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:19.442928 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:19.443627 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:19.443636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:19.443969 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 10:50:19.444216 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 10:50:19.444604 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260927 10:50:19.447269 14677 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33611
I20260927 10:50:19.447539 14677 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:50:19.448215 14677 heartbeater.cc:511] Master 127.6.31.190:33611 requested a full tablet report, sending...
I20260927 10:50:19.449960 14486 ts_manager.cc:194] Registered new tserver with Master: 5d8750335a2a4466882ef9777d7413c6 (127.6.31.129:43569)
I20260927 10:50:19.451340 14486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:46661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:50:19.469893 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33611
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:19.581405 14682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:19.581647 14682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:19.581689 14682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:19.585722 14682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:19.585778 14682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:19.585839 14682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:19.591020 14682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:19.592082 14682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:19.593092 14682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:19.602803 14690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:19.602732 14687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:19.602742 14688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:19.602751 14682 server_base.cc:1034] running on GCE node
I20260927 10:50:19.603557 14682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:19.604187 14682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:19.605374 14682 hybrid_clock.cc:648] HybridClock initialized: now 1790506219605362 us; error 26 us; skew 500 ppm
Sep 27 10:50:19 dist-test-slave-c60x krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:19.608103 14682 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:50:19.609009 14682 webserver.cc:460] Webserver started at http://127.6.31.130:38583/ using document root <none> and password file <none>
I20260927 10:50:19.609555 14682 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:19.609642 14682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:19.609859 14682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:19.611900 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "a3d37ae5c4004ccda924c6ace3cb0be4"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "58676600eda673ada1215129c3c5f916"
server_key_iv: "3fc998e67975ac78868494e88906c160"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.612479 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "a3d37ae5c4004ccda924c6ace3cb0be4"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "58676600eda673ada1215129c3c5f916"
server_key_iv: "3fc998e67975ac78868494e88906c160"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.616209 14682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:19.618829 14697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.620096 14682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:19.620240 14682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "a3d37ae5c4004ccda924c6ace3cb0be4"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "58676600eda673ada1215129c3c5f916"
server_key_iv: "3fc998e67975ac78868494e88906c160"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.620358 14682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:19.639252 14682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:19.642552 14682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:19.642776 14682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:19.643448 14682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:19.644269 14682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:19.644339 14682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.644411 14682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:19.644470 14682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.653126 14682 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:40823
I20260927 10:50:19.653141 14810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:40823 every 8 connection(s)
I20260927 10:50:19.653981 14682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:19.656391 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 14682
I20260927 10:50:19.656524 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:50:19.656844 6270 external_mini_cluster.cc:1428] Setting key 724d4c2ac78c59878b0b7b03e9efd33c
Sep 27 10:50:19 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
I20260927 10:50:19.669384 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:19.655780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:45703 (local address 127.6.31.190:33611)
0927 10:50:19.655939 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:19.655944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:19.656811 (+ 867us) server_negotiation.cc:407] Connection header received
0927 10:50:19.658234 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:19.658237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:19.658287 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:19.658359 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:19.659666 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.660299 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.661461 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.661711 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.661841 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:19.664750 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:19.664755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:19.664760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:19.664795 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:19.666766 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:19.667489 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:19.667492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:19.667494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:19.667601 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:19.667988 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:19.667994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:19.667999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:19.668263 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:19.668372 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:19.669082 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 10:50:19.669209 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:50:19.670387 14813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:19.656219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:45703)
0927 10:50:19.656651 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:19.656702 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:19.657978 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:19.658587 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:19.658593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:19.658946 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:19.659512 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:19.659525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.660427 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.660431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:19.661330 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:19.661340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.661903 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.661910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:19.662137 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:19.662833 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:19.662848 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:19.664578 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:19.666909 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:19.666915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:19.666936 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:19.667324 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:19.667723 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:19.667726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:19.667728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:19.667874 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:19.668540 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:19.668550 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:19.668823 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 10:50:19.669013 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 10:50:19.669347 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":123,"threads_started":1}
I20260927 10:50:19.672384 14811 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33611
I20260927 10:50:19.672649 14811 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:19.673254 14811 heartbeater.cc:511] Master 127.6.31.190:33611 requested a full tablet report, sending...
I20260927 10:50:19.674474 14486 ts_manager.cc:194] Registered new tserver with Master: a3d37ae5c4004ccda924c6ace3cb0be4 (127.6.31.130:40823)
I20260927 10:50:19.675156 14486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:45703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:50:19.687320 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33611
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:19.796666 14816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:19.796932 14816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:19.797012 14816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:19.801102 14816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:19.801163 14816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:19.801262 14816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:19.806790 14816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:19.807894 14816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:19.808907 14816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:19.818415 14821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:19.818429 14824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:19.818425 14822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:19.818480 14816 server_base.cc:1034] running on GCE node
I20260927 10:50:19.819161 14816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:19.819803 14816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:19.820966 14816 hybrid_clock.cc:648] HybridClock initialized: now 1790506219820954 us; error 25 us; skew 500 ppm
Sep 27 10:50:19 dist-test-slave-c60x krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:19.823810 14816 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:50:19.824787 14816 webserver.cc:460] Webserver started at http://127.6.31.131:41905/ using document root <none> and password file <none>
I20260927 10:50:19.825392 14816 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:19.825476 14816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:19.825696 14816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:19.827944 14816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "17d47260027c413a905426192f805535"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "563891259cd2e09a838e7e40af397882"
server_key_iv: "5a2d325fb34ac6b051a5a54df82dda97"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.828507 14816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "17d47260027c413a905426192f805535"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "563891259cd2e09a838e7e40af397882"
server_key_iv: "5a2d325fb34ac6b051a5a54df82dda97"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.832325 14816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 10:50:19.834903 14831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.836267 14816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:50:19.836401 14816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "17d47260027c413a905426192f805535"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "563891259cd2e09a838e7e40af397882"
server_key_iv: "5a2d325fb34ac6b051a5a54df82dda97"
server_key_version: "encryptionkey@0"
I20260927 10:50:19.836515 14816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:19.849902 14816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:19.852850 14816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:19.853070 14816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:19.853677 14816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:19.854561 14816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:19.854630 14816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.854697 14816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:19.854743 14816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:19.863262 14816 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:35197
I20260927 10:50:19.863292 14944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:35197 every 8 connection(s)
I20260927 10:50:19.864181 14816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:50:19.873646 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 14816
I20260927 10:50:19.873822 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:50:19.874193 6270 external_mini_cluster.cc:1428] Setting key 7c12bb0fb6f8cab0a9a4546a851352a8
Sep 27 10:50:19 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:50:19.879072 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:19.866127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:39443 (local address 127.6.31.190:33611)
0927 10:50:19.866336 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:19.866340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:19.867099 (+ 759us) server_negotiation.cc:407] Connection header received
0927 10:50:19.868539 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:19.868542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:19.868599 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:19.868683 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:19.869874 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.870418 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.871765 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.872022 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.872150 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:19.875048 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:19.875051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:19.875054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:19.875119 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:19.877023 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:19.877559 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:19.877562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:19.877563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:19.877656 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:19.878037 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:19.878040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:19.878041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:19.878239 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:19.878322 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:19.878852 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 10:50:19.878955 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260927 10:50:19.879987 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:19.866517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:39443)
0927 10:50:19.866940 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:19.866980 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:19.868338 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:19.868852 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:19.868859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:19.869265 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:19.869726 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:19.869738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.870543 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.870548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:19.871485 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:19.871494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:19.872181 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:19.872185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:19.872367 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:19.873021 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:19.873038 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:19.874811 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:19.877135 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:19.877140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:19.877154 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:19.877429 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:19.877780 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:19.877782 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:19.877784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:19.877934 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:19.878362 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:19.878368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:19.878611 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 10:50:19.878784 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 10:50:19.879044 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260927 10:50:19.881055 14945 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33611
I20260927 10:50:19.881309 14945 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:19.881724 14945 heartbeater.cc:511] Master 127.6.31.190:33611 requested a full tablet report, sending...
I20260927 10:50:19.882643 14486 ts_manager.cc:194] Registered new tserver with Master: 17d47260027c413a905426192f805535 (127.6.31.131:35197)
I20260927 10:50:19.883222 14486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:39443
I20260927 10:50:19.888386 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab.
Entry for principal oryx/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab.
WARNING: no policy specified for oryx/127.6.31.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab.
Entry for principal oryx/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab.
WARNING: no policy specified for oryx/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab.
Entry for principal oryx/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab.
I20260927 10:50:19.964020 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 14548
I20260927 10:50:19.969816 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 14682
I20260927 10:50:19.975013 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 14816
I20260927 10:50:19.980127 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 14454
I20260927 10:50:19.986529 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33611
--webserver_interface=127.6.31.190
--webserver_port=46555
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:33611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:20.101457 14954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:20.101722 14954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:20.101791 14954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:20.105870 14954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:20.105930 14954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:20.105949 14954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:20.105966 14954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:20.106009 14954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:20.111135 14954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:33611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=46555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:20.112115 14954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:20.113113 14954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:20.121557 14960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:20.121595 14954 server_base.cc:1034] running on GCE node
W20260927 10:50:20.121546 14959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:20.121546 14962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:20.122404 14954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:20.123613 14954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:20.124804 14954 hybrid_clock.cc:648] HybridClock initialized: now 1790506220124784 us; error 29 us; skew 500 ppm
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506220, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:20.127835 14954 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:50:20.128749 14954 webserver.cc:460] Webserver started at http://127.6.31.190:46555/ using document root <none> and password file <none>
I20260927 10:50:20.129317 14954 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:20.129423 14954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:20.133253 14954 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:50:20.135468 14969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.136668 14954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:20.136804 14954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "7682808124c040b7ab5b1ce01c727fcc"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "8af4061c2c4410e40c61176f93c39f2b"
server_key_iv: "f5b6c53721987ea375f37067b6ce2e5a"
server_key_version: "encryptionkey@0"
I20260927 10:50:20.137364 14954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:20.148201 14954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:20.151167 14954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:20.151362 14954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:20.158618 14954 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:33611
I20260927 10:50:20.158682 15021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:33611 every 8 connection(s)
I20260927 10:50:20.159789 14954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:20.162758 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 14954
I20260927 10:50:20.163321 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:43569
--local_ip_for_outbound_sockets=127.6.31.129
--tserver_master_addrs=127.6.31.190:33611
--webserver_port=45411
--webserver_interface=127.6.31.129
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.31.129
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:50:20.164485 15022 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:50:20.166716 15022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: Bootstrap starting.
I20260927 10:50:20.183212 15022 log.cc:824] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:20.189574 15022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:50:20.189988 15022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: Bootstrap complete.
I20260927 10:50:20.194137 15022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } }
I20260927 10:50:20.194521 15022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7682808124c040b7ab5b1ce01c727fcc, State: Initialized, Role: FOLLOWER
I20260927 10:50:20.195016 15022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } }
I20260927 10:50:20.195201 15022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:20.195287 15022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:20.195387 15022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:50:20.197300 15022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } }
I20260927 10:50:20.197643 15022 leader_election.cc:302] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7682808124c040b7ab5b1ce01c727fcc; no voters:
I20260927 10:50:20.197970 15022 leader_election.cc:288] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:50:20.198063 15026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:50:20.198297 15026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [term 2 LEADER]: Becoming Leader. State: Replica: 7682808124c040b7ab5b1ce01c727fcc, State: Running, Role: LEADER
I20260927 10:50:20.198648 15026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } }
I20260927 10:50:20.199223 15022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:20.199615 15027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7682808124c040b7ab5b1ce01c727fcc. Latest consensus state: current_term: 2 leader_uuid: "7682808124c040b7ab5b1ce01c727fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } } }
I20260927 10:50:20.199738 15027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:20.199956 15028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7682808124c040b7ab5b1ce01c727fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7682808124c040b7ab5b1ce01c727fcc" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33611 } } }
I20260927 10:50:20.200067 15028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:20.201421 15030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:20.204527 15030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:20.205157 15030 catalog_manager.cc:1259] Loaded cluster ID: ef8c45c0789a4377b82cb187ec4ece00
I20260927 10:50:20.205217 15030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:20.207566 15030 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:20.208225 15030 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7682808124c040b7ab5b1ce01c727fcc: Loaded TSK: 0
I20260927 10:50:20.209404 15030 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:50:20.278913 15024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:20.279333 15024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:20.279431 15024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:20.283696 15024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:20.283798 15024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:20.283893 15024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:20.290244 15024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.129.keytab
--principal=oryx/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:43569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=45411
--tserver_master_addrs=127.6.31.190:33611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:20.291388 15024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:20.292425 15024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:20.302703 15049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:20.302729 15048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:20.302718 15051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:20.303289 15024 server_base.cc:1034] running on GCE node
I20260927 10:50:20.303714 15024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:20.304410 15024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:20.305614 15024 hybrid_clock.cc:648] HybridClock initialized: now 1790506220305574 us; error 67 us; skew 500 ppm
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506220, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:20.308550 15024 init.cc:377] Logged in from keytab as oryx/127.6.31.129@KRBTEST.COM (short username oryx)
I20260927 10:50:20.309796 15024 webserver.cc:460] Webserver started at http://127.6.31.129:45411/ using document root <none> and password file <none>
I20260927 10:50:20.310449 15024 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:20.310529 15024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:20.314877 15024 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:50:20.317134 15058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.318631 15024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:50:20.318782 15024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "5d8750335a2a4466882ef9777d7413c6"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "a9e9f8e3d7b1a9fb029ba7c09f2b942f"
server_key_iv: "677aedfc01fcf36351d7096b71fa8b94"
server_key_version: "encryptionkey@0"
I20260927 10:50:20.319420 15024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:20.342767 15024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:20.346215 15024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:20.346436 15024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:20.347067 15024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:20.347994 15024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:20.348058 15024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.348124 15024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:20.348151 15024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.357111 15024 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:43569
I20260927 10:50:20.357132 15171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:43569 every 8 connection(s)
I20260927 10:50:20.358147 15024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:20.360443 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 15024
I20260927 10:50:20.360985 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:40823
--local_ip_for_outbound_sockets=127.6.31.130
--tserver_master_addrs=127.6.31.190:33611
--webserver_port=38583
--webserver_interface=127.6.31.130
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.31.130
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:20.373956 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.360515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:58543)
0927 10:50:20.361067 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:20.361119 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:20.362591 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:20.363881 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:20.363892 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:20.364410 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:20.365091 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.365113 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.366429 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.366435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.368056 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.368072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.368877 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.368883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.369115 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.369831 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:20.369858 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:20.372900 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":123,"threads_started":1}
W20260927 10:50:20.373945 15174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:20.360408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:58543 (local address 127.6.31.190:33611)
0927 10:50:20.360922 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.360942 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.361381 (+ 439us) server_negotiation.cc:407] Connection header received
0927 10:50:20.363011 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.363023 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.363467 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.363734 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.365289 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.366285 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.368221 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.368697 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.368835 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.373135 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:58543: BlockingRecv error: recv got EOF from 127.6.31.129:58543 (error 108)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":129,"threads_started":1}
W20260927 10:50:20.374534 15172 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:33611 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.31.190:33611: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:20.381227 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.375156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:38335)
0927 10:50:20.375295 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:20.375310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:20.375401 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:20.375672 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:20.375676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:20.375867 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:20.375986 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.375992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.376836 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.376840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.377799 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.377814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.378356 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.378360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.378456 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.378970 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:20.378985 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:20.381078 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:50:20.381433 15174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:20.375221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:38335 (local address 127.6.31.190:33611)
0927 10:50:20.375339 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.375342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.375356 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:50:20.375519 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.375521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.375567 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.375638 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.376193 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.376729 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.377946 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.378170 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.378330 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.381284 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:38335: BlockingRecv error: recv got EOF from 127.6.31.129:38335 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:20.386920 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.381944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:38957)
0927 10:50:20.382136 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:20.382151 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:20.382232 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:20.382463 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:20.382466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:20.382629 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:20.382699 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.382705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.383499 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.383505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.384019 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.384026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.384454 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.384457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.384535 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.385094 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:20.385105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:20.386791 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124}
W20260927 10:50:20.387032 15174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:20.382123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:38957 (local address 127.6.31.190:33611)
0927 10:50:20.382240 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.382244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.382260 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:50:20.382308 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.382313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.382362 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.382448 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.382808 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.383395 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.384135 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.384346 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.384453 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.386911 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:38957: BlockingRecv error: recv got EOF from 127.6.31.129:38957 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 10:50:20.387151 15172 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:50:20.476816 15176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:20.477119 15176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:20.477198 15176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:20.481172 15176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:20.481276 15176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:20.481374 15176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:20.486543 15176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.130.keytab
--principal=oryx/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:40823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=38583
--tserver_master_addrs=127.6.31.190:33611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:20.487641 15176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:20.488641 15176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:20.498202 15181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:20.498337 15176 server_base.cc:1034] running on GCE node
W20260927 10:50:20.498528 15182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:20.498214 15184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:20.499260 15176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:20.499976 15176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:20.501157 15176 hybrid_clock.cc:648] HybridClock initialized: now 1790506220501140 us; error 30 us; skew 500 ppm
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506220, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:20.503970 15176 init.cc:377] Logged in from keytab as oryx/127.6.31.130@KRBTEST.COM (short username oryx)
I20260927 10:50:20.505026 15176 webserver.cc:460] Webserver started at http://127.6.31.130:38583/ using document root <none> and password file <none>
I20260927 10:50:20.505669 15176 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:20.505786 15176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:20.509727 15176 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 10:50:20.511827 15191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.513018 15176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:50:20.513159 15176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "a3d37ae5c4004ccda924c6ace3cb0be4"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "58676600eda673ada1215129c3c5f916"
server_key_iv: "3fc998e67975ac78868494e88906c160"
server_key_version: "encryptionkey@0"
I20260927 10:50:20.513702 15176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:20.541697 15176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:20.544539 15176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:20.544742 15176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:20.545327 15176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:20.546156 15176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:20.546234 15176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.546307 15176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:20.546353 15176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.555291 15176 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:40823
I20260927 10:50:20.555318 15304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:40823 every 8 connection(s)
I20260927 10:50:20.556249 15176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:20.557861 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 15176
I20260927 10:50:20.558337 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:35197
--local_ip_for_outbound_sockets=127.6.31.131
--tserver_master_addrs=127.6.31.190:33611
--webserver_port=41905
--webserver_interface=127.6.31.131
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.31.131
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:50:20.567131 15174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:20.558125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:44467 (local address 127.6.31.190:33611)
0927 10:50:20.558259 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.558262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.559300 (+ 1038us) server_negotiation.cc:407] Connection header received
0927 10:50:20.560556 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.560560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.560625 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.560729 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.561839 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.562355 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.563530 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.563749 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.563860 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.566981 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:44467: BlockingRecv error: recv got EOF from 127.6.31.130:44467 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:50:20.567726 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.558625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:44467)
0927 10:50:20.559144 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:20.559200 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:20.560353 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:20.560907 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:20.560913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:20.561250 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:20.561689 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.561701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.562476 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.562480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.563413 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.563422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.563893 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.563897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.564058 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.564670 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:20.564685 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:20.566870 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":131,"threads_started":1}
W20260927 10:50:20.568143 15305 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:33611 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.31.190:33611: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:20.574278 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.568586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:55181)
0927 10:50:20.568742 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:20.568755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:20.568819 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:20.569046 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:20.569048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:20.569212 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:20.569327 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.569333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.569986 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.569989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.570883 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.570892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.571474 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.571477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.571577 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.572088 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:20.572122 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:20.574147 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260927 10:50:20.574402 15174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:20.568673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:55181 (local address 127.6.31.190:33611)
0927 10:50:20.568796 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.568799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.568811 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:50:20.568904 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.568906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.568942 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.569005 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.569428 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.569892 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.570997 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.571328 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.571461 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.574281 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:55181: BlockingRecv error: recv got EOF from 127.6.31.130:55181 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:20.580250 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.575061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:38029)
0927 10:50:20.575331 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:20.575350 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:20.575435 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:20.575644 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:20.575647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:20.575799 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:20.575880 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.575885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.576561 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.576564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.577414 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.577423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.577815 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.577819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.577901 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.578327 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:20.578340 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:20.580111 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180}
W20260927 10:50:20.580397 15174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:20.575170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:38029 (local address 127.6.31.190:33611)
0927 10:50:20.575295 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.575298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.575396 (+ 98us) server_negotiation.cc:407] Connection header received
0927 10:50:20.575511 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.575513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.575545 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.575607 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.575977 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.576467 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.577520 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.577710 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.577783 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.580256 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:38029: BlockingRecv error: recv got EOF from 127.6.31.130:38029 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 10:50:20.580576 15305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:50:20.671996 15308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:20.672297 15308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:20.672374 15308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:20.676373 15308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:20.676470 15308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:20.676587 15308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:20.681731 15308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/krb5kdc/oryx_127.6.31.131.keytab
--principal=oryx/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:35197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=41905
--tserver_master_addrs=127.6.31.190:33611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:20.682798 15308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:20.683815 15308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:20.693161 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:20.693288 15308 server_base.cc:1034] running on GCE node
W20260927 10:50:20.693243 15313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:20.693161 15314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:20.693893 15308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:20.694528 15308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:20.695711 15308 hybrid_clock.cc:648] HybridClock initialized: now 1790506220695687 us; error 47 us; skew 500 ppm
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506220, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:20.698421 15308 init.cc:377] Logged in from keytab as oryx/127.6.31.131@KRBTEST.COM (short username oryx)
I20260927 10:50:20.699347 15308 webserver.cc:460] Webserver started at http://127.6.31.131:41905/ using document root <none> and password file <none>
I20260927 10:50:20.700053 15308 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:20.700158 15308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:20.704054 15308 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:50:20.705998 15323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.707234 15308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:20.707320 15308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "17d47260027c413a905426192f805535"
format_stamp: "Formatted at 2026-09-27 10:50:19 on dist-test-slave-c60x"
server_key: "563891259cd2e09a838e7e40af397882"
server_key_iv: "5a2d325fb34ac6b051a5a54df82dda97"
server_key_version: "encryptionkey@0"
I20260927 10:50:20.707849 15308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:20.723455 15308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:20.726552 15308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:20.726750 15308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:20.727415 15308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:20.728219 15308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:20.728263 15308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.728327 15308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:20.728359 15308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:20.737289 15308 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:35197
I20260927 10:50:20.737309 15436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:35197 every 8 connection(s)
I20260927 10:50:20.738308 15308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:50:20.744863 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 15308
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:50:20.749969 15174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:20.740188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:49605 (local address 127.6.31.190:33611)
0927 10:50:20.740352 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.740357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.741225 (+ 868us) server_negotiation.cc:407] Connection header received
0927 10:50:20.742509 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.742512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.742563 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.742655 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.743850 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.744445 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.745857 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.746125 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.746202 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.749842 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:49605: BlockingRecv error: recv got EOF from 127.6.31.131:49605 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 10:50:20.750864 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.740571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:49605)
0927 10:50:20.741073 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:20.741115 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:20.742325 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:20.742825 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:20.742831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:20.743197 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:20.743688 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.743701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.744588 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.744591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.745709 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.745720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.746398 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.746402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.746579 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.747215 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:20.747232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:20.749691 (+ 2459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":113,"threads_started":1}
W20260927 10:50:20.751516 15437 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:33611 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.31.190:33611: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
I20260927 10:50:20.755475 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.745920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.6.31.190:33611)
0927 10:50:20.746202 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.746206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.746217 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:50:20.746249 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.746251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.746295 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.746385 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.747223 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.747727 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.748676 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.749042 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.749166 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.751058 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:20.751065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:20.751085 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:20.751139 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:20.753264 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:20.753694 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:20.753702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:20.753711 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:20.753869 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:20.754246 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:20.754249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:20.754250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:20.754608 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:20.754802 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:20.755048 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 10:50:20.755346 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:20.757750 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.751997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:56141)
0927 10:50:20.752294 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:20.752310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:20.752383 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:20.752641 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:20.752643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:20.752772 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:20.752860 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.752866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.753780 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.753783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.754485 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.754494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.755046 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.755050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.755170 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.755699 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:20.755715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:20.757634 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 10:50:20.757905 15174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:20.752068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:56141 (local address 127.6.31.190:33611)
0927 10:50:20.752189 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.752192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.752370 (+ 178us) server_negotiation.cc:407] Connection header received
0927 10:50:20.752447 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.752450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.752495 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.752590 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.752975 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.753677 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.754592 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.754924 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.755019 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.757770 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:56141: BlockingRecv error: recv got EOF from 127.6.31.131:56141 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:20 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:20.762764 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:20.758392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:54833)
0927 10:50:20.758510 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:20.758524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:20.758591 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:20.758819 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:20.758820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:20.758934 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:20.758998 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.759003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.759708 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.759710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.760238 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:20.760246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.760632 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.760635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:20.760687 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.761165 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:20.761175 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:20.762680 (+ 1505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:50:20.762897 15174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:20.758451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:54833 (local address 127.6.31.190:33611)
0927 10:50:20.758632 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:20.758637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:20.758651 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:50:20.758683 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:20.758686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:20.758727 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:20.758790 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:20.759123 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.759616 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.760338 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:20.760539 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:20.760616 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:20.762791 (+ 2175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:54833: BlockingRecv error: recv got EOF from 127.6.31.131:54833 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260927 10:50:20.762992 15437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:50:21 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:21 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:21.392905 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:21.387780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:57083)
0927 10:50:21.388041 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:21.388054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:21.388186 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:21.388438 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:21.388440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:21.388608 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:21.388719 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:21.388724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.389418 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.389420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:21.389968 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:21.389975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.390367 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.390369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:21.390439 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:21.391019 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:21.391031 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:21.392789 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
W20260927 10:50:21.393055 15443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:21.387851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:57083 (local address 127.6.31.190:33611)
0927 10:50:21.388077 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:21.388084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:21.388161 (+ 77us) server_negotiation.cc:407] Connection header received
0927 10:50:21.388282 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:21.388284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:21.388330 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:21.388407 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:21.388816 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.389331 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.390062 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.390275 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.390347 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:21.392923 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:57083: BlockingRecv error: recv got EOF from 127.6.31.129:57083 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":57,"threads_started":1}
Sep 27 10:50:21 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:21 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:21.586597 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:21.581177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:34617)
0927 10:50:21.581414 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:21.581430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:21.581534 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:21.581824 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:21.581827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:21.582061 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:21.582180 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:21.582186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.582944 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.582951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:21.583597 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:21.583606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.584055 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.584058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:21.584151 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:21.584620 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:21.584632 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:21.586426 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
W20260927 10:50:21.586649 15443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:21.581237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:34617 (local address 127.6.31.190:33611)
0927 10:50:21.581366 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:21.581371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:21.581497 (+ 126us) server_negotiation.cc:407] Connection header received
0927 10:50:21.581639 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:21.581641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:21.581686 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:21.581793 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:21.582304 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.582838 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.583725 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.583947 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.584052 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:21.586544 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:34617: BlockingRecv error: recv got EOF from 127.6.31.130:34617 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 10:50:21 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:21 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:21.769551 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:21.763761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:39969)
0927 10:50:21.764070 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:21.764083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:21.764172 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:21.764409 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:21.764411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:21.764576 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:21.764664 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:21.764669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.765417 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.765420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:21.766004 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:21.766013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.766513 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.766517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:21.766614 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:21.767266 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:21.767281 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:21.769371 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
W20260927 10:50:21.769591 15443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:21.764073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:39969 (local address 127.6.31.190:33611)
0927 10:50:21.764197 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:21.764202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:21.764222 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:50:21.764259 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:21.764261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:21.764302 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:21.764371 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:21.764771 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.765298 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.766142 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:21.766369 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:21.766496 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:21.769488 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:39969: BlockingRecv error: recv got EOF from 127.6.31.131:39969 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:50:22 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:22 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:22.399709 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:22.393815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:38495)
0927 10:50:22.394086 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:22.394103 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:22.394214 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:22.394546 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:22.394552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:22.394765 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:22.394861 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:22.394867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.395700 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.395703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:22.396296 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:22.396304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.396796 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.396799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:22.396887 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:22.397424 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:22.397436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:22.399527 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
W20260927 10:50:22.399885 15447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:22.393864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:38495 (local address 127.6.31.190:33611)
0927 10:50:22.394146 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:22.394151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:22.394166 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:50:22.394340 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:22.394343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:22.394399 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:22.394504 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:22.395003 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.395593 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.396436 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.396680 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.396768 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:22.399684 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:38495: BlockingRecv error: recv got EOF from 127.6.31.129:38495 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":106,"threads_started":1}
Sep 27 10:50:22 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:22 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:22.593070 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:22.587548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:35331)
0927 10:50:22.587826 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:22.587838 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:22.587970 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:22.588274 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:22.588277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:22.588472 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:22.588555 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:22.588561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.589330 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.589336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:22.589878 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:22.589885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.590338 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.590342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:22.590422 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:22.590903 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:22.590913 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:22.592947 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
W20260927 10:50:22.593232 15447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:22.587606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:35331 (local address 127.6.31.190:33611)
0927 10:50:22.587773 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:22.587777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:22.587910 (+ 133us) server_negotiation.cc:407] Connection header received
0927 10:50:22.588066 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:22.588069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:22.588115 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:22.588196 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:22.588700 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.589223 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.589994 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.590233 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.590338 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:22.593094 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:35331: BlockingRecv error: recv got EOF from 127.6.31.130:35331 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 10:50:22 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:22 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:22.775784 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:22.770504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:43009)
0927 10:50:22.770791 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:22.770806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:22.770912 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:22.771181 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:22.771183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:22.771360 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:22.771453 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:22.771457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.772188 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.772191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:22.772770 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:22.772779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.773172 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.773175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:22.773242 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:22.773741 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:22.773752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:22.775621 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":123,"threads_started":1}
W20260927 10:50:22.775851 15447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:22.770610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:43009 (local address 127.6.31.190:33611)
0927 10:50:22.770763 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:22.770766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:22.770882 (+ 116us) server_negotiation.cc:407] Connection header received
0927 10:50:22.770991 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:22.770994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:22.771037 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:22.771130 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:22.771544 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.772066 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.772869 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:22.773068 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:22.773140 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:22.775750 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:43009: BlockingRecv error: recv got EOF from 127.6.31.131:43009 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 10:50:23 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:23 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:23.411562 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:23.400669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:57075)
0927 10:50:23.400931 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:23.400947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:23.401048 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:23.401311 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:23.401313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:23.401485 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:23.401626 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:23.401635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.402422 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.402425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:23.402996 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:23.403003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.403552 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.403556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:23.403623 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:23.409440 (+ 5817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:23.409452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:23.411436 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
W20260927 10:50:23.411742 15451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:23.400736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:57075 (local address 127.6.31.190:33611)
0927 10:50:23.401018 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:23.401022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:23.401036 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:50:23.401116 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:23.401118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:23.401159 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:23.401246 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:23.401748 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.402300 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.403131 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.403407 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.403486 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:23.411599 (+ 8113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:57075: BlockingRecv error: recv got EOF from 127.6.31.129:57075 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
Sep 27 10:50:23 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:23 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:23.600257 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:23.593867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:42477)
0927 10:50:23.594199 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:23.594211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:23.594307 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:23.594590 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:23.594592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:23.594755 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:23.594848 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:23.594852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.596109 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.596112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:23.596683 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:23.596692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.597347 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.597349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:23.597408 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:23.597940 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:23.597951 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:23.599988 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":89,"threads_started":1}
W20260927 10:50:23.600282 15451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:23.593979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:42477 (local address 127.6.31.190:33611)
0927 10:50:23.594153 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:23.594160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:23.594285 (+ 125us) server_negotiation.cc:407] Connection header received
0927 10:50:23.594392 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:23.594396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:23.594461 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:23.594568 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:23.594962 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.595987 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.596800 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.597238 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.597363 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:23.600152 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:42477: BlockingRecv error: recv got EOF from 127.6.31.130:42477 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 10:50:23 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:23 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:23.782238 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:23.776746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:54795)
0927 10:50:23.777062 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:23.777079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:23.777203 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:23.777466 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:23.777469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:23.777649 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:23.777759 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:23.777765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.778622 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.778628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:23.779211 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:23.779219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.779663 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.779666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:23.779756 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:23.780310 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:23.780322 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:23.782069 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":76,"threads_started":1}
W20260927 10:50:23.782342 15451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:23.776900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:54795 (local address 127.6.31.190:33611)
0927 10:50:23.777064 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:23.777068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:23.777176 (+ 108us) server_negotiation.cc:407] Connection header received
0927 10:50:23.777289 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:23.777292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:23.777338 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:23.777402 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:23.777904 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.778511 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.779337 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:23.779553 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:23.779663 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:23.782192 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:54795: BlockingRecv error: recv got EOF from 127.6.31.131:54795 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 10:50:24 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:24 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:24.418183 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:24.412463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:35111)
0927 10:50:24.412747 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:24.412764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:24.412874 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:24.413174 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:24.413177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:24.413412 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:24.413505 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:24.413511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.414284 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.414288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:24.414828 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:24.414836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.415380 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.415384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:24.415476 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:24.415964 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:24.415977 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:24.418068 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
W20260927 10:50:24.418360 15455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:24.412509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:35111 (local address 127.6.31.190:33611)
0927 10:50:24.412846 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:24.412851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:24.412864 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:50:24.412994 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:24.412999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:24.413047 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:24.413149 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:24.413627 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.414166 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.414949 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.415259 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.415365 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:24.418218 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:35111: BlockingRecv error: recv got EOF from 127.6.31.129:35111 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":61,"threads_started":1}
Sep 27 10:50:24 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:24 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:24.606366 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:24.601022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:37841)
0927 10:50:24.601313 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:24.601326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:24.601449 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:24.601714 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:24.601716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:24.601943 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:24.602028 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:24.602034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.602811 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.602815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:24.603447 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:24.603456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.603902 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.603905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:24.603988 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:24.604469 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:24.604480 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:24.606212 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
W20260927 10:50:24.606484 15455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:24.601094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:37841 (local address 127.6.31.190:33611)
0927 10:50:24.601234 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:24.601240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:24.601397 (+ 157us) server_negotiation.cc:407] Connection header received
0927 10:50:24.601552 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:24.601556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:24.601609 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:24.601694 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:24.602143 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.602691 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.603566 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.603789 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.603897 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:24.606380 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:37841: BlockingRecv error: recv got EOF from 127.6.31.130:37841 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 10:50:24 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:24 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:24.789430 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:24.783140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:53893)
0927 10:50:24.783446 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:24.783463 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:24.783582 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:24.783912 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:24.783915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:24.784166 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:24.784289 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:24.784295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.785232 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.785235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:24.785816 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:24.785824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.786293 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.786297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:24.786366 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:24.786905 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:24.786918 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:24.789313 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
W20260927 10:50:24.789578 15455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:24.783213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:53893 (local address 127.6.31.190:33611)
0927 10:50:24.783393 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:24.783398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:24.783573 (+ 175us) server_negotiation.cc:407] Connection header received
0927 10:50:24.783710 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:24.783713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:24.783779 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:24.783877 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:24.784407 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.785139 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.785925 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:24.786132 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:24.786209 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:24.789459 (+ 3250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:53893: BlockingRecv error: recv got EOF from 127.6.31.131:53893 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 10:50:25 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:25 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:25.424485 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:25.418997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:45485)
0927 10:50:25.419292 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:25.419306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:25.419397 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:25.419628 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:25.419630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:25.419828 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:25.419919 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:25.419924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.420661 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.420664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:25.421235 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:25.421242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.421676 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.421679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:25.421759 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:25.422248 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:25.422262 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:25.424354 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":121,"threads_started":1}
W20260927 10:50:25.424546 15459 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:25.419057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:45485 (local address 127.6.31.190:33611)
0927 10:50:25.419307 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:25.419310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:25.419367 (+ 57us) server_negotiation.cc:407] Connection header received
0927 10:50:25.419474 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:25.419476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:25.419520 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:25.419599 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:25.420016 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.420547 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.421338 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.421555 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.421631 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:25.424442 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:45485: BlockingRecv error: recv got EOF from 127.6.31.129:45485 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
Sep 27 10:50:25 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:25 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:25.613562 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:25.607288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:41969)
0927 10:50:25.607706 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:25.607721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:25.607859 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:25.608151 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:25.608154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:25.608367 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:25.608453 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:25.608459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.609280 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.609286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:25.609890 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:25.609899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.610356 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.610360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:25.610443 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:25.610978 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:25.610990 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:25.613393 (+ 2403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":111,"threads_started":1}
W20260927 10:50:25.613623 15459 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:25.607448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:41969 (local address 127.6.31.190:33611)
0927 10:50:25.607623 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:25.607648 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:25.607808 (+ 160us) server_negotiation.cc:407] Connection header received
0927 10:50:25.607967 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:25.607971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:25.608034 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:25.608124 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:25.608566 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.609110 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.610009 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.610240 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.610350 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:25.613517 (+ 3167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:41969: BlockingRecv error: recv got EOF from 127.6.31.130:41969 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 10:50:25 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:25 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:25.795784 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:25.790320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:51123)
0927 10:50:25.790628 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:25.790646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:25.790752 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:25.790978 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:25.790979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:25.791382 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:25.791478 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:25.791484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.792200 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.792203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:25.792765 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:25.792773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.793179 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.793181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:25.793241 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:25.793821 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:25.793833 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:25.795668 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
W20260927 10:50:25.795919 15459 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:25.790471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:51123 (local address 127.6.31.190:33611)
0927 10:50:25.790625 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:25.790629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:25.790717 (+ 88us) server_negotiation.cc:407] Connection header received
0927 10:50:25.790829 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:25.790832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:25.790876 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:25.790942 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:25.791578 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.792091 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.792873 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:25.793080 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:25.793158 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:25.795796 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:51123: BlockingRecv error: recv got EOF from 127.6.31.131:51123 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 10:50:26 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:26 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:26.430536 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:26.425407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:43233)
0927 10:50:26.425694 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:26.425709 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:26.425802 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:26.426035 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:26.426037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:26.426244 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:26.426335 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:26.426341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.427066 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.427069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:26.427676 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:26.427685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.428104 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.428107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:26.428167 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:26.428663 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:26.428674 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:26.430423 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
W20260927 10:50:26.430721 15463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:26.425468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:43233 (local address 127.6.31.190:33611)
0927 10:50:26.425733 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:26.425737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:26.425752 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:50:26.425883 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:26.425885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:26.425927 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:26.426002 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:26.426428 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.426956 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.427790 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.428007 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.428085 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:26.430551 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:43233: BlockingRecv error: recv got EOF from 127.6.31.129:43233 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":108,"threads_started":1}
Sep 27 10:50:26 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:26 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:26.620085 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:26.614326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:47975)
0927 10:50:26.614602 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:26.614614 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:26.614764 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:26.615022 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:26.615024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:26.615259 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:26.615350 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:26.615356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.616105 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.616111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:26.616793 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:26.616802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.617243 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.617247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:26.617329 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:26.617886 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:26.617897 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:26.619918 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
W20260927 10:50:26.620193 15463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:26.614381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:47975 (local address 127.6.31.190:33611)
0927 10:50:26.614543 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:26.614548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:26.614685 (+ 137us) server_negotiation.cc:407] Connection header received
0927 10:50:26.614859 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:26.614863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:26.614916 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:26.615004 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:26.615462 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.615984 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.616906 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.617129 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.617235 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:26.620073 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:47975: BlockingRecv error: recv got EOF from 127.6.31.130:47975 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 10:50:26 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:26 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:26.802337 15465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:26.796682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:43285)
0927 10:50:26.797011 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:26.797024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:26.797167 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:26.797438 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:26.797441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:26.797664 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:26.797754 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:26.797760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.798522 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.798528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:26.799180 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:26.799193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.799643 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.799646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:26.799727 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:26.800307 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:26.800327 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:26.802190 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
W20260927 10:50:26.802495 15463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:26.796797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:43285 (local address 127.6.31.190:33611)
0927 10:50:26.796962 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:26.796966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:26.797107 (+ 141us) server_negotiation.cc:407] Connection header received
0927 10:50:26.797269 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:26.797273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:26.797328 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:26.797417 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:26.797878 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.798414 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.799308 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:26.799529 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:26.799644 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:26.802340 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:43285: BlockingRecv error: recv got EOF from 127.6.31.131:43285 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 10:50:27 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:27 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:27.436422 15466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:27.431247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:52465)
0927 10:50:27.431559 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:27.431573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:27.431660 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:27.431914 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:27.431916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:27.432091 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:27.432181 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:27.432185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.432923 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.432925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:27.433485 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:27.433493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.433905 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.433907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:27.433965 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:27.434525 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:27.434537 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:27.436309 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":66,"threads_started":1}
W20260927 10:50:27.436575 15467 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:27.431345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:52465 (local address 127.6.31.190:33611)
0927 10:50:27.431606 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:27.431610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:27.431622 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:50:27.431742 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:27.431745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:27.431803 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:27.431877 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:27.432274 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.432824 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.433585 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.433813 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.433888 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:27.436445 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:52465: BlockingRecv error: recv got EOF from 127.6.31.129:52465 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
Sep 27 10:50:27 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:27 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:27.626247 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:27.620777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:54565)
0927 10:50:27.621073 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:27.621086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:27.621184 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:27.621433 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:27.621435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:27.621607 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:27.621698 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:27.621703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.622410 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.622412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:27.622980 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:27.622986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.623467 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.623469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:27.623531 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:27.624024 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:27.624035 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:27.626082 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
W20260927 10:50:27.626396 15467 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:27.620857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:54565 (local address 127.6.31.190:33611)
0927 10:50:27.621016 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:27.621023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:27.621159 (+ 136us) server_negotiation.cc:407] Connection header received
0927 10:50:27.621287 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:27.621289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:27.621335 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:27.621402 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:27.621791 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.622324 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.623127 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.623371 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.623451 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:27.626250 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:54565: BlockingRecv error: recv got EOF from 127.6.31.130:54565 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 10:50:27 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:27 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:27.809800 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:27.803145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:58933)
0927 10:50:27.803506 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:27.803519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:27.803612 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:27.803919 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:27.803922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:27.804094 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:27.804186 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:27.804191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.805426 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.805428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:27.806035 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:27.806044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.806760 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.806763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:27.806827 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:27.807476 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:27.807488 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:27.809655 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
W20260927 10:50:27.810032 15467 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:27.803201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:58933 (local address 127.6.31.190:33611)
0927 10:50:27.803355 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:27.803359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:27.803600 (+ 241us) server_negotiation.cc:407] Connection header received
0927 10:50:27.803688 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:27.803694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:27.803763 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:27.803868 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:27.804316 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.805298 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.806157 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:27.806619 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:27.806736 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:27.809863 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:58933: BlockingRecv error: recv got EOF from 127.6.31.131:58933 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 10:50:28 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:28 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:28.455943 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:28.437355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:41915)
0927 10:50:28.437736 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:28.437754 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:28.437881 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:28.438182 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:28.438184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:28.438450 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:28.438549 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:28.438555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.439381 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.439384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:28.439937 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:28.439945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.440387 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.440390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:28.440498 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:28.453506 (+ 13008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:28.453521 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:28.455813 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":86,"threads_started":1}
W20260927 10:50:28.456278 15471 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:28.437471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:41915 (local address 127.6.31.190:33611)
0927 10:50:28.437773 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:28.437777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:28.437810 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:50:28.437994 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:28.437999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:28.438056 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:28.438168 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:28.438651 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.439286 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.440060 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.440286 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.440374 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:28.455978 (+ 15604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:41915: BlockingRecv error: recv got EOF from 127.6.31.129:41915 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
Sep 27 10:50:28 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:28 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:28.632578 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:28.627147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:59113)
0927 10:50:28.627420 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:28.627433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:28.627528 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:28.627788 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:28.627790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:28.627965 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:28.628054 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:28.628059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.628986 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.628989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:28.629538 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:28.629546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.629965 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.629968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:28.630030 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:28.630497 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:28.630509 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:28.632461 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":103,"threads_started":1}
W20260927 10:50:28.632705 15471 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:28.627227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:59113 (local address 127.6.31.190:33611)
0927 10:50:28.627392 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:28.627395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:28.627500 (+ 105us) server_negotiation.cc:407] Connection header received
0927 10:50:28.627607 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:28.627610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:28.627653 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:28.627731 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:28.628160 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.628878 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.629650 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.629868 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.629947 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:28.632591 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:59113: BlockingRecv error: recv got EOF from 127.6.31.130:59113 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 10:50:28 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:28 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:28.816360 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:28.810709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:34941)
0927 10:50:28.810996 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:28.811010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:28.811160 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:28.811463 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:28.811466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:28.811647 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:28.811756 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:28.811761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.812649 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.812652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:28.813219 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:28.813228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.813660 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.813662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:28.813732 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:28.814216 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:28.814228 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:28.816226 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
W20260927 10:50:28.816641 15471 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:28.810762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:34941 (local address 127.6.31.190:33611)
0927 10:50:28.810928 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:28.810932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:28.811104 (+ 172us) server_negotiation.cc:407] Connection header received
0927 10:50:28.811267 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:28.811269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:28.811354 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:28.811430 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:28.811876 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.812539 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.813328 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:28.813540 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:28.813640 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:28.816403 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:34941: BlockingRecv error: recv got EOF from 127.6.31.131:34941 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 10:50:29 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:29 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:29.462345 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:29.456850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:44321)
0927 10:50:29.457124 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:29.457137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:29.457232 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:29.457534 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:29.457537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:29.457718 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:29.457839 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:29.457852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.458618 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.458637 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:29.459249 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:29.459258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.459683 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.459685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:29.459745 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:29.460393 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:29.460405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:29.462226 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
W20260927 10:50:29.462577 15475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:29.456918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:44321 (local address 127.6.31.190:33611)
0927 10:50:29.457173 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:29.457177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:29.457190 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:50:29.457325 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:29.457328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:29.457373 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:29.457458 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:29.457975 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.458510 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.459361 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.459581 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.459666 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:29.462410 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:44321: BlockingRecv error: recv got EOF from 127.6.31.129:44321 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
Sep 27 10:50:29 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:29 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:29.638697 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:29.633454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:60031)
0927 10:50:29.633731 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:29.633744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:29.633884 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:29.634122 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:29.634124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:29.634301 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:29.634392 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:29.634397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.635172 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.635175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:29.635730 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:29.635738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.636159 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.636162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:29.636225 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:29.636767 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:29.636778 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:29.638583 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
W20260927 10:50:29.638922 15475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:29.633539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:60031 (local address 127.6.31.190:33611)
0927 10:50:29.633684 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:29.633692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:29.633847 (+ 155us) server_negotiation.cc:407] Connection header received
0927 10:50:29.633976 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:29.633978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:29.634023 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:29.634091 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:29.634487 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.635008 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.635841 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.636054 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.636149 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:29.638778 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:60031: BlockingRecv error: recv got EOF from 127.6.31.130:60031 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 10:50:29 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:29 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:29.823820 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:29.817235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:54713)
0927 10:50:29.817630 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:29.817646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:29.817806 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:29.818123 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:29.818126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:29.818379 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:29.818478 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:29.818484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.819456 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.819461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:29.820109 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:29.820118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.820640 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.820644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:29.820733 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:29.821378 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:29.821392 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:29.823600 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":150,"threads_started":1}
W20260927 10:50:29.823891 15475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:29.817306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:54713 (local address 127.6.31.190:33611)
0927 10:50:29.817478 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:29.817483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:29.817753 (+ 270us) server_negotiation.cc:407] Connection header received
0927 10:50:29.817929 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:29.817933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:29.817995 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:29.818178 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:29.818610 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.819328 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.820252 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:29.820504 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:29.820630 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:29.823768 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:54713: BlockingRecv error: recv got EOF from 127.6.31.131:54713 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 10:50:30 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:30 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:30.468382 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:30.463174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:43613)
0927 10:50:30.463457 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:30.463475 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:30.463575 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:30.463854 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:30.463857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:30.464039 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:30.464134 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:30.464139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.464870 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.464873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:30.465427 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:30.465434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.465835 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.465837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:30.465896 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:30.466410 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:30.466421 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:30.468247 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":69,"threads_started":1}
W20260927 10:50:30.468566 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:30.463327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:43613 (local address 127.6.31.190:33611)
0927 10:50:30.463553 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:30.463556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:30.463569 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:50:30.463657 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:30.463660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:30.463700 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:30.463781 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:30.464239 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.464753 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.465521 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.465736 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.465829 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:30.468414 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:43613: BlockingRecv error: recv got EOF from 127.6.31.129:43613 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
Sep 27 10:50:30 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:30 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:30.644716 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:30.639536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:57037)
0927 10:50:30.639790 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:30.639803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:30.639893 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:30.640166 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:30.640168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:30.640342 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:30.640430 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:30.640435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.641284 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.641286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:30.641901 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:30.641910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.642301 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.642303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:30.642364 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:30.642823 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:30.642833 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:30.644594 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
W20260927 10:50:30.644847 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:30.639608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:57037 (local address 127.6.31.190:33611)
0927 10:50:30.639750 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:30.639754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:30.639903 (+ 149us) server_negotiation.cc:407] Connection header received
0927 10:50:30.639998 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:30.640003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:30.640060 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:30.640138 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:30.640538 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.641180 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.641985 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.642211 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.642289 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:30.644735 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:57037: BlockingRecv error: recv got EOF from 127.6.31.130:57037 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 10:50:30 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:30 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:30.829916 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:30.824646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:59567)
0927 10:50:30.824944 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:30.824961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:30.825064 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:30.825279 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:30.825281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:30.825460 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:30.825550 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:30.825555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.826275 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.826277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:30.826890 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:30.826898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.827371 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.827373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:30.827435 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:30.827991 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:30.828003 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:30.829792 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
W20260927 10:50:30.830083 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:30.824791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:59567 (local address 127.6.31.190:33611)
0927 10:50:30.824945 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:30.824948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:30.825036 (+ 88us) server_negotiation.cc:407] Connection header received
0927 10:50:30.825136 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:30.825138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:30.825179 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:30.825248 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:30.825647 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.826170 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.826999 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:30.827230 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:30.827348 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:30.829950 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:59567: BlockingRecv error: recv got EOF from 127.6.31.131:59567 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 10:50:31 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:31 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:31.475271 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:31.469183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:45191)
0927 10:50:31.469559 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:31.469581 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:31.469700 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:31.470156 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:31.470159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:31.470341 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:31.470436 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:31.470441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.471312 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.471314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:31.471857 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:31.471865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.472318 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.472320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:31.472380 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:31.472952 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:31.472964 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:31.474998 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":155,"threads_started":1}
W20260927 10:50:31.475555 15483 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:31.469311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:45191 (local address 127.6.31.190:33611)
0927 10:50:31.469565 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:31.469584 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:31.469641 (+ 57us) server_negotiation.cc:407] Connection header received
0927 10:50:31.469882 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:31.469885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:31.469972 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:31.470075 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:31.470556 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.471150 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.471977 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.472198 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.472305 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:31.475419 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:45191: BlockingRecv error: recv got EOF from 127.6.31.129:45191 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
Sep 27 10:50:31 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:31 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:31.650853 15484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:31.645538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:49667)
0927 10:50:31.645799 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:31.645817 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:31.645923 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:31.646118 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:31.646120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:31.646297 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:31.646387 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:31.646392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.647142 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.647146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:31.647738 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:31.647748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.648192 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.648196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:31.648298 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:31.648808 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:31.648819 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:31.650724 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
W20260927 10:50:31.650998 15483 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:31.645620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:49667 (local address 127.6.31.190:33611)
0927 10:50:31.645760 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:31.645766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:31.645896 (+ 130us) server_negotiation.cc:407] Connection header received
0927 10:50:31.646002 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:31.646004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:31.646048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:31.646110 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:31.646478 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.646997 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.647854 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.648064 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.648141 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:31.650867 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:49667: BlockingRecv error: recv got EOF from 127.6.31.130:49667 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:50:31 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:31 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:31.836175 15485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:31.830712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:51953)
0927 10:50:31.830988 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:31.831004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:31.831126 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:31.831414 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:31.831417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:31.831619 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:31.831707 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:31.831713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.832465 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.832469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:31.833018 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:31.833026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.833488 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.833492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:31.833586 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:31.834141 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:31.834153 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:31.835990 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":109,"threads_started":1}
W20260927 10:50:31.836212 15483 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:31.830928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:51953 (local address 127.6.31.190:33611)
0927 10:50:31.831053 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:31.831057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:31.831071 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:50:31.831236 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:31.831241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:31.831299 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:31.831389 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:31.831828 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.832357 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.833134 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:31.833373 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:31.833475 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:31.836115 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:51953: BlockingRecv error: recv got EOF from 127.6.31.131:51953 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 10:50:32 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:32 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:32.482185 15486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:32.476264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:38509)
0927 10:50:32.476559 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:32.476572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:32.476668 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:32.476937 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:32.476939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:32.477113 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:32.477203 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:32.477208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.478012 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.478015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:32.478716 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:32.478725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.479434 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.479437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:32.479523 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:32.480046 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:32.480058 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:32.482011 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
W20260927 10:50:32.482364 15487 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:32.476393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:38509 (local address 127.6.31.190:33611)
0927 10:50:32.476625 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:32.476629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:32.476643 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:50:32.476775 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:32.476777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:32.476823 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:32.476901 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:32.477345 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.477899 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.478882 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.479295 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.479418 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:32.482143 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:38509: BlockingRecv error: recv got EOF from 127.6.31.129:38509 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":57,"threads_started":1}
Sep 27 10:50:32 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:32 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:32.657936 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:32.651853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:57263)
0927 10:50:32.652217 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:32.652231 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:32.652386 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:32.652653 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:32.652655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:32.652874 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:32.652974 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:32.652980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.653740 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.653747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:32.654304 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:32.654312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.654770 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.654774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:32.654857 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:32.655456 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:32.655473 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:32.657816 (+ 2343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":75,"threads_started":1}
W20260927 10:50:32.658089 15487 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:32.651922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:57263 (local address 127.6.31.190:33611)
0927 10:50:32.652100 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:32.652104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:32.652321 (+ 217us) server_negotiation.cc:407] Connection header received
0927 10:50:32.652490 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:32.652493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:32.652547 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:32.652657 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:32.653098 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.653624 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.654424 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.654655 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.654764 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:32.657939 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:57263: BlockingRecv error: recv got EOF from 127.6.31.130:57263 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 10:50:32 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:32 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:32.842679 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:32.836990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:59877)
0927 10:50:32.837340 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:32.837353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:32.837448 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:32.837760 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:32.837762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:32.837944 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:32.838047 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:32.838053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.838826 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.838828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:32.839432 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:32.839440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.839867 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.839869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:32.839928 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:32.840434 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:32.840446 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:32.842509 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":158,"threads_started":1}
W20260927 10:50:32.842751 15487 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:32.837124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:59877 (local address 127.6.31.190:33611)
0927 10:50:32.837274 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:32.837278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:32.837423 (+ 145us) server_negotiation.cc:407] Connection header received
0927 10:50:32.837531 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:32.837533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:32.837588 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:32.837683 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:32.838165 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.838688 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.839533 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:32.839757 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:32.839846 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:32.842638 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:59877: BlockingRecv error: recv got EOF from 127.6.31.131:59877 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 10:50:33 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:33 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:33.492563 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:33.483144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:59105)
0927 10:50:33.483421 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:33.483435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:33.483577 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:33.483888 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:33.483891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:33.484095 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:33.484185 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:33.484191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.484971 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.484973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:33.485527 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:33.485534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.485953 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.485956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:33.486023 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:33.490335 (+ 4312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:33.490349 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:33.492438 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
W20260927 10:50:33.492800 15491 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:33.483183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:59105 (local address 127.6.31.190:33611)
0927 10:50:33.483475 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:33.483479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:33.483516 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:50:33.483683 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:33.483686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:33.483744 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:33.483837 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:33.484290 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.484854 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.485624 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.485837 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.485910 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:33.492649 (+ 6739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:59105: BlockingRecv error: recv got EOF from 127.6.31.129:59105 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
Sep 27 10:50:33 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:33 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:33.664113 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:33.658721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:50427)
0927 10:50:33.659009 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:33.659022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:33.659170 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:33.659452 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:33.659454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:33.659703 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:33.659848 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:33.659856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.660563 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.660565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:33.661114 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:33.661121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.661548 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.661550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:33.661608 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:33.662080 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:33.662091 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:33.663945 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
W20260927 10:50:33.664180 15491 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:33.658835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:50427 (local address 127.6.31.190:33611)
0927 10:50:33.658978 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:33.658982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:33.659112 (+ 130us) server_negotiation.cc:407] Connection header received
0927 10:50:33.659273 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:33.659276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:33.659335 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:33.659425 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:33.659954 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.660473 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.661215 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.661455 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.661531 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:33.664075 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:50427: BlockingRecv error: recv got EOF from 127.6.31.130:50427 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:50:33 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:33 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:33.849540 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:33.843528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:48093)
0927 10:50:33.843876 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:33.843891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:33.844052 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:33.844442 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:33.844445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:33.844710 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:33.844827 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:33.844836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.845686 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.845692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:33.846340 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:33.846349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.846826 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.846829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:33.846913 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:33.847480 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:33.847501 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:33.849379 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":84,"threads_started":1}
W20260927 10:50:33.849653 15491 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:33.843653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:48093 (local address 127.6.31.190:33611)
0927 10:50:33.843838 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:33.843842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:33.843990 (+ 148us) server_negotiation.cc:407] Connection header received
0927 10:50:33.844264 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:33.844268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:33.844323 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:33.844414 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:33.844926 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.845557 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.846474 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:33.846710 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:33.846823 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:33.849542 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:48093: BlockingRecv error: recv got EOF from 127.6.31.131:48093 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 10:50:34 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:34 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:34.498826 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:34.493374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:49579)
0927 10:50:34.493664 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:34.493684 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:34.493803 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:34.494051 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:34.494053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:34.494234 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:34.494326 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:34.494331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.495067 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.495069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:34.495678 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:34.495686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.496124 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.496126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:34.496187 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:34.496675 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:34.496686 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:34.498642 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
W20260927 10:50:34.498992 15495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:34.493540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:49579 (local address 127.6.31.190:33611)
0927 10:50:34.493839 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:34.493842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:34.493857 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:50:34.493893 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:34.493895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:34.493937 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:34.494014 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:34.494430 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.494974 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.495801 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.496017 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.496094 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:34.498855 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:49579: BlockingRecv error: recv got EOF from 127.6.31.129:49579 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":75,"threads_started":1}
Sep 27 10:50:34 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:34 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:34.670799 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:34.665023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:54821)
0927 10:50:34.665294 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:34.665307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:34.665411 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:34.665660 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:34.665662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:34.665865 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:34.665960 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:34.665965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.666812 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.666815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:34.667460 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:34.667470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.667927 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.667930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:34.667997 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:34.668614 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:34.668626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:34.670675 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
W20260927 10:50:34.671000 15495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:34.665097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:54821 (local address 127.6.31.190:33611)
0927 10:50:34.665236 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:34.665240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:34.665395 (+ 155us) server_negotiation.cc:407] Connection header received
0927 10:50:34.665500 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:34.665502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:34.665551 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:34.665622 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:34.666088 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.666674 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.667582 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.667814 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.667905 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:34.670833 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:54821: BlockingRecv error: recv got EOF from 127.6.31.130:54821 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:50:34 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:34 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.131@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:34.856303 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:34.850418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.131:56259)
0927 10:50:34.850753 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:34.850771 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:34.850895 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:34.851204 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:34.851207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:34.851458 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:34.851551 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:34.851557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.852424 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.852430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:34.853158 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:34.853167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.853686 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.853689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:34.853753 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:34.854279 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:34.854290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:34.856188 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":125,"threads_started":1}
W20260927 10:50:34.856506 15495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:34.850588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:56259 (local address 127.6.31.190:33611)
0927 10:50:34.850745 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:34.850750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:34.850858 (+ 108us) server_negotiation.cc:407] Connection header received
0927 10:50:34.850984 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:34.850988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:34.851042 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:34.851179 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:34.851693 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.852315 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.853344 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:34.853580 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:34.853665 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:34.856369 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.131:56259: BlockingRecv error: recv got EOF from 127.6.31.131:56259 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 10:50:35 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:35 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.129@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:35.505534 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:35.499872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.129:33361)
0927 10:50:35.500135 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:35.500149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:35.500272 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:35.500635 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:35.500638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:35.500827 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:35.500920 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:35.500925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.501789 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.501792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:35.502420 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:35.502428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.502859 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.502862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:35.502924 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:35.503485 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:35.503497 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:35.505404 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
W20260927 10:50:35.505674 15499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:35.500023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:33361 (local address 127.6.31.190:33611)
0927 10:50:35.500351 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:35.500354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:35.500367 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:50:35.500401 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:35.500403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:35.500483 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:35.500568 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:35.501069 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.501685 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.502538 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.502754 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.502839 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:35.505540 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.129:33361: BlockingRecv error: recv got EOF from 127.6.31.129:33361 (error 108)
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":207,"threads_started":1}
Sep 27 10:50:35 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:50:35 dist-test-slave-c60x krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.31.130@KRBTEST.COM for oryx/127.6.31.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:50:35.677590 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:35.671858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33611 (local address 127.6.31.130:56857)
0927 10:50:35.672146 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:35.672159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:35.672258 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:35.672526 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:35.672554 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:35.672810 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:35.672906 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:35.672912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.673824 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.673827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:35.674542 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:35.674551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.674970 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.674973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:35.675035 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:35.675588 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:35.675600 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:35.677462 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.31.190:33611: Server oryx/127.6.31.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":116,"threads_started":1}
W20260927 10:50:35.677712 15499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:35.671986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:56857 (local address 127.6.31.190:33611)
0927 10:50:35.672134 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:35.672137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:35.672226 (+ 89us) server_negotiation.cc:407] Connection header received
0927 10:50:35.672334 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:35.672337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:35.672395 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:35.672497 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:35.673033 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.673701 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.674648 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.674867 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.674946 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:35.677598 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.31.130:56857: BlockingRecv error: recv got EOF from 127.6.31.130:56857 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:50:35.754690 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 15024
I20260927 10:50:35.760346 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 15176
I20260927 10:50:35.765851 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 15308
I20260927 10:50:35.771620 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 14954
2026-09-27T10:50:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17969 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T10:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:35Z Disabled control of system clock
I20260927 10:50:35.802399 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46187
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:44305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:46187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:50:35.913384 15504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:35.913688 15504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:35.913725 15504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:35.917837 15504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:35.917905 15504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:35.917922 15504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:35.917937 15504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:35.917953 15504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:35.922824 15504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:46187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:35.924019 15504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:35.925422 15504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:35.934604 15510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:35.934628 15509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:35.934599 15512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:35.934798 15504 server_base.cc:1034] running on GCE node
I20260927 10:50:35.935650 15504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:35.936882 15504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:35.938076 15504 hybrid_clock.cc:648] HybridClock initialized: now 1790506235938065 us; error 29 us; skew 500 ppm
I20260927 10:50:35.940056 15504 webserver.cc:460] Webserver started at http://127.6.31.190:34895/ using document root <none> and password file <none>
I20260927 10:50:35.940653 15504 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:35.940745 15504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:35.940977 15504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:35.943061 15504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "2e15629746d748b5a3261605281ac00d"
format_stamp: "Formatted at 2026-09-27 10:50:35 on dist-test-slave-c60x"
server_key: "194e222fd451b2667eb701cd9740fd26"
server_key_iv: "030f9fac60ac484fef518748d05e2e59"
server_key_version: "encryptionkey@0"
I20260927 10:50:35.943699 15504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "2e15629746d748b5a3261605281ac00d"
format_stamp: "Formatted at 2026-09-27 10:50:35 on dist-test-slave-c60x"
server_key: "194e222fd451b2667eb701cd9740fd26"
server_key_iv: "030f9fac60ac484fef518748d05e2e59"
server_key_version: "encryptionkey@0"
I20260927 10:50:35.947455 15504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:35.949935 15518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:35.951370 15504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:35.951514 15504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "2e15629746d748b5a3261605281ac00d"
format_stamp: "Formatted at 2026-09-27 10:50:35 on dist-test-slave-c60x"
server_key: "194e222fd451b2667eb701cd9740fd26"
server_key_iv: "030f9fac60ac484fef518748d05e2e59"
server_key_version: "encryptionkey@0"
I20260927 10:50:35.951635 15504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:35.969237 15504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:35.969964 15504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:35.970198 15504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:35.977243 15504 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:46187
I20260927 10:50:35.977253 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:46187 every 8 connection(s)
I20260927 10:50:35.978192 15504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:35.981468 15571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:35.987265 15571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d: Bootstrap starting.
I20260927 10:50:35.988473 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 15504
I20260927 10:50:35.988586 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:35.988881 6270 external_mini_cluster.cc:1428] Setting key 33640805fe7b984c549d2be7bd6ad70c
I20260927 10:50:35.990352 15571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:35.991168 15571 log.cc:824] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:35.993537 15571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d: No bootstrap required, opened a new log
I20260927 10:50:35.997996 15571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e15629746d748b5a3261605281ac00d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46187 } }
I20260927 10:50:35.998255 15571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:35.998348 15571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e15629746d748b5a3261605281ac00d, State: Initialized, Role: FOLLOWER
I20260927 10:50:35.998409 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:35.990065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.6.31.190:46187)
0927 10:50:35.990719 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:35.990734 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:35.990770 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:50:35.991561 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:35.991586 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:35.992009 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:35.992311 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:35.992651 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.993619 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.994861 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:35.995423 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:35.995551 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:35.996444 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:35.996448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:35.996460 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:35.996846 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:35.996904 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:35.996923 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:35.997003 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:35.997204 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 10:50:35.997595 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":138,"threads_started":1}
I20260927 10:50:35.998878 15571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e15629746d748b5a3261605281ac00d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46187 } }
I20260927 10:50:35.999039 15571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:35.999157 15571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:35.999256 15571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:36.000345 15571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e15629746d748b5a3261605281ac00d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46187 } }
I20260927 10:50:36.000757 15571 leader_election.cc:302] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e15629746d748b5a3261605281ac00d; no voters:
I20260927 10:50:36.001106 15571 leader_election.cc:288] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:36.001245 15576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:36.001502 15576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [term 1 LEADER]: Becoming Leader. State: Replica: 2e15629746d748b5a3261605281ac00d, State: Running, Role: LEADER
I20260927 10:50:36.001941 15576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e15629746d748b5a3261605281ac00d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46187 } }
I20260927 10:50:36.002465 15571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:36.003566 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e15629746d748b5a3261605281ac00d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e15629746d748b5a3261605281ac00d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46187 } } }
I20260927 10:50:36.003693 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:36.004088 15585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:36.004067 15577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e15629746d748b5a3261605281ac00d. Latest consensus state: current_term: 1 leader_uuid: "2e15629746d748b5a3261605281ac00d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e15629746d748b5a3261605281ac00d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46187 } } }
I20260927 10:50:36.004791 15577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:36.007200 15585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:36.012516 15585 catalog_manager.cc:1347] Generated new cluster ID: 5063b7926f9845a69f090100443c16a2
I20260927 10:50:36.012583 15585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:36.029846 15585 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:36.031126 15585 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:36.040184 15585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e15629746d748b5a3261605281ac00d: Generated new TSK 0
I20260927 10:50:36.040961 15585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:50:36.046216 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46187
--builtin_ntp_servers=127.6.31.148:44305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 10:50:36.172739 15595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:36.172992 15595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:36.173030 15595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:36.177129 15595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:36.177186 15595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:36.177273 15595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:36.181949 15595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:36.183049 15595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:36.184459 15595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:36.194999 15603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:36.195008 15600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:36.194999 15601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:36.195060 15595 server_base.cc:1034] running on GCE node
I20260927 10:50:36.195715 15595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:36.196382 15595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:36.197571 15595 hybrid_clock.cc:648] HybridClock initialized: now 1790506236197537 us; error 48 us; skew 500 ppm
I20260927 10:50:36.199506 15595 webserver.cc:460] Webserver started at http://127.6.31.129:43655/ using document root <none> and password file <none>
I20260927 10:50:36.200048 15595 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:36.200109 15595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:36.200344 15595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:36.202412 15595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "1d427caba2d24dfa97b7fe5c119dffaf"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "59261e65aa913d1d8ce4653a962e9af4"
server_key_iv: "2213a5421ba6b7c77df4b1c16af9e16d"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.203159 15595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "1d427caba2d24dfa97b7fe5c119dffaf"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "59261e65aa913d1d8ce4653a962e9af4"
server_key_iv: "2213a5421ba6b7c77df4b1c16af9e16d"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.207067 15595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 10:50:36.209817 15609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.211297 15595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 10:50:36.211428 15595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "1d427caba2d24dfa97b7fe5c119dffaf"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "59261e65aa913d1d8ce4653a962e9af4"
server_key_iv: "2213a5421ba6b7c77df4b1c16af9e16d"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.211570 15595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:36.230224 15595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:36.230952 15595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:36.231225 15595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:36.231827 15595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:36.232769 15595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:36.232839 15595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.232919 15595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:36.232980 15595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.242017 15595 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:34073
I20260927 10:50:36.242030 15722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:34073 every 8 connection(s)
I20260927 10:50:36.242920 15595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:36.243705 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 15595
I20260927 10:50:36.243822 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:50:36.244102 6270 external_mini_cluster.cc:1428] Setting key 730c344f80bb1737a6ce4f10bc04b0de
I20260927 10:50:36.246394 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46187
--builtin_ntp_servers=127.6.31.148:44305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:50:36.253618 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:36.245861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:56501 (local address 127.6.31.190:46187)
0927 10:50:36.246021 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:36.246026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:36.247274 (+ 1248us) server_negotiation.cc:407] Connection header received
0927 10:50:36.248635 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:36.248638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:36.248692 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:36.248771 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:36.249636 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.250260 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.251479 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.251731 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.251852 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:36.253071 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:36.253075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:36.253077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:36.253090 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:36.253109 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:36.253116 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:36.253193 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:36.253402 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 10:50:36.253464 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:50:36.254596 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:36.246349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46187 (local address 127.6.31.129:56501)
0927 10:50:36.247111 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:36.247164 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:36.248424 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:36.248936 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:36.248943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:36.249023 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:36.249499 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:36.249510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.250400 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.250408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:36.251359 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:36.251368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.251893 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.251897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:36.252064 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:36.252762 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:36.252764 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:36.252835 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:36.252904 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:36.253206 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:36.253216 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:36.253229 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 10:50:36.253343 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 10:50:36.253655 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":149,"threads_started":1}
I20260927 10:50:36.256304 15723 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46187
I20260927 10:50:36.256580 15723 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:36.257153 15723 heartbeater.cc:511] Master 127.6.31.190:46187 requested a full tablet report, sending...
I20260927 10:50:36.258855 15535 ts_manager.cc:194] Registered new tserver with Master: 1d427caba2d24dfa97b7fe5c119dffaf (127.6.31.129:34073)
I20260927 10:50:36.260475 15535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.129:56501
W20260927 10:50:36.360739 15726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:36.361029 15726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:36.361068 15726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:36.365154 15726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:36.365208 15726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:36.365262 15726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:36.370074 15726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:36.371229 15726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:36.372565 15726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:36.382246 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:36.382318 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:36.382249 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:36.382726 15726 server_base.cc:1034] running on GCE node
I20260927 10:50:36.383183 15726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:36.383880 15726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:36.385105 15726 hybrid_clock.cc:648] HybridClock initialized: now 1790506236385097 us; error 41 us; skew 500 ppm
I20260927 10:50:36.387029 15726 webserver.cc:460] Webserver started at http://127.6.31.130:44637/ using document root <none> and password file <none>
I20260927 10:50:36.387679 15726 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:36.387744 15726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:36.387923 15726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:36.390039 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "d86e2e2cbce94f6d8e3bbc4dc198a5a2"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "370221490e2f5390d89d752da9a0bbea"
server_key_iv: "c3bd5f5d28cca3c03179ea15353a38e3"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.390678 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "d86e2e2cbce94f6d8e3bbc4dc198a5a2"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "370221490e2f5390d89d752da9a0bbea"
server_key_iv: "c3bd5f5d28cca3c03179ea15353a38e3"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.394670 15726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 10:50:36.397383 15740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.398671 15726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:50:36.398813 15726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "d86e2e2cbce94f6d8e3bbc4dc198a5a2"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "370221490e2f5390d89d752da9a0bbea"
server_key_iv: "c3bd5f5d28cca3c03179ea15353a38e3"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.398965 15726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:36.427480 15726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:36.428224 15726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:36.428431 15726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:36.429049 15726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:36.430007 15726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:36.430079 15726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.430152 15726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:36.430195 15726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.439248 15726 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:38599
I20260927 10:50:36.439270 15853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:38599 every 8 connection(s)
I20260927 10:50:36.440223 15726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:36.443023 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 15726
I20260927 10:50:36.443149 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:50:36.443444 6270 external_mini_cluster.cc:1428] Setting key 1d280b63240579baf2b75f07838a91c0
I20260927 10:50:36.445607 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46187
--builtin_ntp_servers=127.6.31.148:44305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:50:36.450536 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:36.442128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:56937 (local address 127.6.31.190:46187)
0927 10:50:36.442265 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:36.442268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:36.443421 (+ 1153us) server_negotiation.cc:407] Connection header received
0927 10:50:36.445102 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:36.445106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:36.445193 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:36.445288 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:36.446438 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.446965 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.448142 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.448539 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.448661 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:36.449926 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:36.449932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:36.449937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:36.449954 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:36.449980 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:36.449993 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:36.450087 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:36.450243 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 10:50:36.450325 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:50:36.451403 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:36.442497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46187 (local address 127.6.31.130:56937)
0927 10:50:36.443250 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:36.443313 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:36.444824 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:36.445514 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:36.445524 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:36.445625 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:36.446304 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:36.446319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.447127 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.447130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:36.448013 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:36.448022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.448712 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.448715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:36.448881 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:36.449607 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:36.449612 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:36.449672 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:36.449755 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:36.450092 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:36.450100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:36.450105 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:50:36.450209 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:50:36.450443 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":122,"threads_started":1}
I20260927 10:50:36.452688 15854 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46187
I20260927 10:50:36.452966 15854 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:36.453671 15854 heartbeater.cc:511] Master 127.6.31.190:46187 requested a full tablet report, sending...
I20260927 10:50:36.455032 15535 ts_manager.cc:194] Registered new tserver with Master: d86e2e2cbce94f6d8e3bbc4dc198a5a2 (127.6.31.130:38599)
I20260927 10:50:36.455724 15535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.130:56937
W20260927 10:50:36.561435 15857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:36.561712 15857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:36.561750 15857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:36.565936 15857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:36.566000 15857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:36.566076 15857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:36.571065 15857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:36.572278 15857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:36.573909 15857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:36.583899 15863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:36.583900 15862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:36.583966 15865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:36.584270 15857 server_base.cc:1034] running on GCE node
I20260927 10:50:36.584656 15857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:36.585368 15857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:36.586549 15857 hybrid_clock.cc:648] HybridClock initialized: now 1790506236586491 us; error 76 us; skew 500 ppm
I20260927 10:50:36.588601 15857 webserver.cc:460] Webserver started at http://127.6.31.131:35067/ using document root <none> and password file <none>
I20260927 10:50:36.589179 15857 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:36.589249 15857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:36.589501 15857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:36.591610 15857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "ea7b29af948746709661a5d5596382ae"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "99760f442e85ef356ddeb89ad41cb381"
server_key_iv: "bfae412fb05f29fdb60823f56cad4b16"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.592166 15857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "ea7b29af948746709661a5d5596382ae"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "99760f442e85ef356ddeb89ad41cb381"
server_key_iv: "bfae412fb05f29fdb60823f56cad4b16"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.595842 15857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:50:36.598331 15871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.599838 15857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:36.599938 15857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "ea7b29af948746709661a5d5596382ae"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "99760f442e85ef356ddeb89ad41cb381"
server_key_iv: "bfae412fb05f29fdb60823f56cad4b16"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.600082 15857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:36.613876 15857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:36.614584 15857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:36.614794 15857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:36.615556 15857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:36.616568 15857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:36.616614 15857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.616679 15857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:36.616724 15857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.625957 15857 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:43875
I20260927 10:50:36.625979 15984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:43875 every 8 connection(s)
I20260927 10:50:36.626924 15857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:50:36.632817 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 15857
I20260927 10:50:36.632944 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:50:36.633189 6270 external_mini_cluster.cc:1428] Setting key b35c256e04afc51f47f492b0fe3699ab
I20260927 10:50:36.637410 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:36.628859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:33151 (local address 127.6.31.190:46187)
0927 10:50:36.629027 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:36.629031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:36.630284 (+ 1253us) server_negotiation.cc:407] Connection header received
0927 10:50:36.632157 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:36.632161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:36.632254 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:36.632371 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:36.633500 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.634106 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.635410 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.635637 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.635750 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:36.636906 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:36.636910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:36.636913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:36.636932 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:36.636950 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:36.636957 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:36.637032 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:36.637207 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 10:50:36.637266 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 10:50:36.638322 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:36.629267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46187 (local address 127.6.31.131:33151)
0927 10:50:36.630112 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:36.630170 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:36.631934 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:36.632549 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:36.632559 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:36.632672 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:36.633347 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:36.633370 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.634238 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.634241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:36.635261 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:36.635275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:36.635791 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:36.635795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:36.636008 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:36.636647 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:36.636649 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:36.636689 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:36.636754 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:36.637045 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:36.637056 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:36.637062 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:50:36.637197 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 10:50:36.637553 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":159,"threads_started":1}
I20260927 10:50:36.639436 15985 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46187
I20260927 10:50:36.639688 15985 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:36.640250 15985 heartbeater.cc:511] Master 127.6.31.190:46187 requested a full tablet report, sending...
I20260927 10:50:36.641255 15535 ts_manager.cc:194] Registered new tserver with Master: ea7b29af948746709661a5d5596382ae (127.6.31.131:43875)
I20260927 10:50:36.641885 15535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.131:33151
I20260927 10:50:36.647185 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:50:36.650385 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:36.649331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.6.31.190:46187)
0927 10:50:36.649478 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:36.649481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:36.649627 (+ 146us) server_negotiation.cc:407] Connection header received
0927 10:50:36.649773 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:36.649776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:36.649820 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:36.649898 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:36.650180 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38134: BlockingRecv error: recv got EOF from 127.0.0.1:38134 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 10:50:36.650492 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 15595
I20260927 10:50:36.656443 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 15726
I20260927 10:50:36.662035 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 15857
I20260927 10:50:36.667415 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 15504
2026-09-27T10:50:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (897 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T10:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:36Z Disabled control of system clock
I20260927 10:50:36.702435 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37143
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:37143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:50:36.818243 15996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:36.818533 15996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:36.818572 15996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:36.822666 15996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:36.822739 15996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:36.822754 15996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:36.822772 15996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:36.822788 15996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:36.827665 15996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:37143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:36.828764 15996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:36.830089 15996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:36.838586 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:36.838677 16002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:36.838584 16004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:36.838999 15996 server_base.cc:1034] running on GCE node
I20260927 10:50:36.839628 15996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:36.840719 15996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:36.841886 15996 hybrid_clock.cc:648] HybridClock initialized: now 1790506236841869 us; error 31 us; skew 500 ppm
I20260927 10:50:36.843879 15996 webserver.cc:460] Webserver started at http://127.6.31.190:44043/ using document root <none> and password file <none>
I20260927 10:50:36.844450 15996 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:36.844511 15996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:36.844671 15996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:36.846695 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "aa23490acb394ff0bcaffcd1b7ddbccf"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "f26992206b1012a105ab9a1e9d580c7a"
server_key_iv: "3108700c5a273977ed04068d55ea5f2b"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.847281 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "aa23490acb394ff0bcaffcd1b7ddbccf"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "f26992206b1012a105ab9a1e9d580c7a"
server_key_iv: "3108700c5a273977ed04068d55ea5f2b"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.850903 15996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:50:36.853310 16010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:36.854585 15996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:50:36.854710 15996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "aa23490acb394ff0bcaffcd1b7ddbccf"
format_stamp: "Formatted at 2026-09-27 10:50:36 on dist-test-slave-c60x"
server_key: "f26992206b1012a105ab9a1e9d580c7a"
server_key_iv: "3108700c5a273977ed04068d55ea5f2b"
server_key_version: "encryptionkey@0"
I20260927 10:50:36.854799 15996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:36.859274 15996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:36.859829 15996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:36.859961 15996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:36.866781 15996 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:37143
I20260927 10:50:36.866858 16062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:37143 every 8 connection(s)
I20260927 10:50:36.867789 15996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:36.868391 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 15996
I20260927 10:50:36.868502 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:36.868813 6270 external_mini_cluster.cc:1428] Setting key d843b80a413a388b2f81b034b7722650
I20260927 10:50:36.871359 16063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:36.875392 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:36.870267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.6.31.190:37143)
0927 10:50:36.871645 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:36.871664 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:36.871715 (+ 51us) server_negotiation.cc:407] Connection header received
0927 10:50:36.872372 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:36.872388 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:36.872757 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:36.873106 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:36.874025 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:36.874029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:36.874041 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:36.874052 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:36.874111 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:36.874126 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:36.874294 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:36.874365 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:50:36.874604 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":741,"threads_started":1}
I20260927 10:50:36.877789 16063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf: Bootstrap starting.
I20260927 10:50:36.880009 16063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:36.880751 16063 log.cc:824] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:36.882865 16063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf: No bootstrap required, opened a new log
I20260927 10:50:36.886547 16063 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa23490acb394ff0bcaffcd1b7ddbccf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37143 } }
I20260927 10:50:36.886808 16063 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:36.886871 16063 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa23490acb394ff0bcaffcd1b7ddbccf, State: Initialized, Role: FOLLOWER
I20260927 10:50:36.887385 16063 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa23490acb394ff0bcaffcd1b7ddbccf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37143 } }
I20260927 10:50:36.887537 16063 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:36.887603 16063 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:36.887698 16063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:36.888801 16063 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa23490acb394ff0bcaffcd1b7ddbccf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37143 } }
I20260927 10:50:36.889133 16063 leader_election.cc:302] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa23490acb394ff0bcaffcd1b7ddbccf; no voters:
I20260927 10:50:36.889451 16063 leader_election.cc:288] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:36.889590 16068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:36.889838 16068 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [term 1 LEADER]: Becoming Leader. State: Replica: aa23490acb394ff0bcaffcd1b7ddbccf, State: Running, Role: LEADER
I20260927 10:50:36.890125 16068 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa23490acb394ff0bcaffcd1b7ddbccf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37143 } }
I20260927 10:50:36.890681 16063 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:36.891829 16069 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa23490acb394ff0bcaffcd1b7ddbccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa23490acb394ff0bcaffcd1b7ddbccf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37143 } } }
I20260927 10:50:36.891865 16070 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa23490acb394ff0bcaffcd1b7ddbccf. Latest consensus state: current_term: 1 leader_uuid: "aa23490acb394ff0bcaffcd1b7ddbccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa23490acb394ff0bcaffcd1b7ddbccf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37143 } } }
I20260927 10:50:36.892091 16070 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:36.892046 16069 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:36.892958 16077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:36.895709 16077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:36.901530 16077 catalog_manager.cc:1347] Generated new cluster ID: ce370ecbc0624bd0ba352ec7d72cc6d6
I20260927 10:50:36.901628 16077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:36.918962 16077 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:36.919366 16077 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:36.925392 16077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa23490acb394ff0bcaffcd1b7ddbccf: Generated new TSK 0
I20260927 10:50:36.926255 16077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:50:36.933486 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37143
--builtin_ntp_servers=127.6.31.148:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:50:37.043483 16087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:37.043780 16087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:37.043851 16087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:37.047894 16087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:37.047976 16087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:37.048069 16087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:37.052894 16087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:37.053956 16087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:37.055325 16087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:37.064960 16092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:37.064950 16093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:37.064970 16095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:37.065117 16087 server_base.cc:1034] running on GCE node
I20260927 10:50:37.065653 16087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:37.066290 16087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:37.067487 16087 hybrid_clock.cc:648] HybridClock initialized: now 1790506237067473 us; error 24 us; skew 500 ppm
I20260927 10:50:37.069448 16087 webserver.cc:460] Webserver started at http://127.6.31.129:40145/ using document root <none> and password file <none>
I20260927 10:50:37.070019 16087 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:37.070078 16087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:37.070317 16087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:37.072391 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "98252ac7790f4166a093ef2c66c5f63b"
format_stamp: "Formatted at 2026-09-27 10:50:37 on dist-test-slave-c60x"
server_key: "7869fbc526894c9935228e530e7722da"
server_key_iv: "8fd8ee78cde00c01b305216070b5d858"
server_key_version: "encryptionkey@0"
I20260927 10:50:37.072947 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "98252ac7790f4166a093ef2c66c5f63b"
format_stamp: "Formatted at 2026-09-27 10:50:37 on dist-test-slave-c60x"
server_key: "7869fbc526894c9935228e530e7722da"
server_key_iv: "8fd8ee78cde00c01b305216070b5d858"
server_key_version: "encryptionkey@0"
I20260927 10:50:37.076702 16087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:50:37.079403 16101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:37.080719 16087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:37.080861 16087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "98252ac7790f4166a093ef2c66c5f63b"
format_stamp: "Formatted at 2026-09-27 10:50:37 on dist-test-slave-c60x"
server_key: "7869fbc526894c9935228e530e7722da"
server_key_iv: "8fd8ee78cde00c01b305216070b5d858"
server_key_version: "encryptionkey@0"
I20260927 10:50:37.080986 16087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:37.085606 16087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:37.086202 16087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:37.086472 16087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:37.087271 16087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:37.088254 16087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:37.088332 16087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:37.088402 16087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:37.088456 16087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:37.098634 16087 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:43625
I20260927 10:50:37.098659 16214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:43625 every 8 connection(s)
I20260927 10:50:37.099756 16087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:37.105955 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:37.101723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:38063 (local address 127.6.31.190:37143)
0927 10:50:37.101886 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:37.101889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:37.102900 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 10:50:37.104279 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:37.104283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:37.104329 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:37.104409 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:37.105475 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:37.105479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:37.105482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:37.105494 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:37.105516 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:37.105532 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:37.105631 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:37.105763 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 10:50:37.105807 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:50:37.106827 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:37.102123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37143 (local address 127.6.31.129:38063)
0927 10:50:37.102754 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:37.102791 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:37.104075 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:37.104566 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:37.104572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:37.104653 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:37.105205 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:37.105206 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:37.105243 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:37.105306 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:37.105614 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:37.105622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:37.105628 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:50:37.105808 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 10:50:37.106039 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
I20260927 10:50:37.108181 16215 heartbeater.cc:347] Connected to a master server at 127.6.31.190:37143
I20260927 10:50:37.108412 16215 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:37.108814 16215 heartbeater.cc:511] Master 127.6.31.190:37143 requested a full tablet report, sending...
I20260927 10:50:37.110066 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 16087
I20260927 10:50:37.110162 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:50:37.110500 6270 external_mini_cluster.cc:1428] Setting key 5243d1ef0ca366b31f08a479245d08f0
I20260927 10:50:37.110482 16027 ts_manager.cc:194] Registered new tserver with Master: 98252ac7790f4166a093ef2c66c5f63b (127.6.31.129:43625)
I20260927 10:50:37.112850 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37143
--builtin_ntp_servers=127.6.31.148:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:50:37.229084 16218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:37.229362 16218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:37.229403 16218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:37.233489 16218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:37.233557 16218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:37.233616 16218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:37.238466 16218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:37.239707 16218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:37.241112 16218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:37.250753 16223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:37.250813 16218 server_base.cc:1034] running on GCE node
W20260927 10:50:37.250825 16224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:37.250753 16226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:37.251438 16218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:37.252101 16218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:37.253315 16218 hybrid_clock.cc:648] HybridClock initialized: now 1790506237253307 us; error 39 us; skew 500 ppm
I20260927 10:50:37.255300 16218 webserver.cc:460] Webserver started at http://127.6.31.130:40333/ using document root <none> and password file <none>
I20260927 10:50:37.255910 16218 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:37.256011 16218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:37.256258 16218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:37.258518 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "141c520ec29d496987d527b8f8234855"
format_stamp: "Formatted at 2026-09-27 10:50:37 on dist-test-slave-c60x"
server_key: "d7ee7a750243940435e9a87e18b4d204"
server_key_iv: "d7bade151671d27cdc87fdc775def2a9"
server_key_version: "encryptionkey@0"
I20260927 10:50:37.259246 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "141c520ec29d496987d527b8f8234855"
format_stamp: "Formatted at 2026-09-27 10:50:37 on dist-test-slave-c60x"
server_key: "d7ee7a750243940435e9a87e18b4d204"
server_key_iv: "d7bade151671d27cdc87fdc775def2a9"
server_key_version: "encryptionkey@0"
I20260927 10:50:37.263387 16218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:50:37.266088 16232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:37.267517 16218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:37.267655 16218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "141c520ec29d496987d527b8f8234855"
format_stamp: "Formatted at 2026-09-27 10:50:37 on dist-test-slave-c60x"
server_key: "d7ee7a750243940435e9a87e18b4d204"
server_key_iv: "d7bade151671d27cdc87fdc775def2a9"
server_key_version: "encryptionkey@0"
I20260927 10:50:37.267787 16218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:37.272320 16218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:37.272921 16218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:37.273102 16218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:37.273635 16218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:37.274557 16218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:37.274626 16218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:37.274703 16218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:37.274744 16218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:37.283346 16218 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:42097
I20260927 10:50:37.283371 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:42097 every 8 connection(s)
I20260927 10:50:37.284353 16218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:37.289655 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 16218
I20260927 10:50:37.289793 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:50:37.290122 6270 external_mini_cluster.cc:1428] Setting key fdc4505f2869be2e1fc38254329ef82e
I20260927 10:50:37.290827 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:37.286365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:42473 (local address 127.6.31.190:37143)
0927 10:50:37.286540 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:37.286554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:37.287672 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 10:50:37.288896 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:37.288898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:37.288936 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:37.289015 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:37.290297 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:37.290301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:37.290303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:37.290314 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:37.290331 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:37.290339 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:37.290431 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:37.290597 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:50:37.290644 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:50:37.291865 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:37.286688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37143 (local address 127.6.31.130:42473)
0927 10:50:37.287481 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:37.287522 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:37.288687 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:37.289162 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:37.289168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:37.289249 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:37.289975 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:37.289978 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:37.290035 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:37.290115 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:37.290438 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:37.290446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:37.290452 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:50:37.290631 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 10:50:37.290857 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260927 10:50:37.292397 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37143
--builtin_ntp_servers=127.6.31.148:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 10:50:37.293052 16346 heartbeater.cc:347] Connected to a master server at 127.6.31.190:37143
I20260927 10:50:37.293277 16346 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:37.293732 16346 heartbeater.cc:511] Master 127.6.31.190:37143 requested a full tablet report, sending...
I20260927 10:50:37.294771 16026 ts_manager.cc:194] Registered new tserver with Master: 141c520ec29d496987d527b8f8234855 (127.6.31.130:42097)
W20260927 10:50:37.406769 16349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:37.407087 16349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:37.407207 16349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:37.411490 16349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:37.411577 16349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:37.411671 16349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:37.416628 16349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:37.417852 16349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:37.419431 16349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:37.429282 16355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:37.429282 16357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:37.429292 16354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:37.429461 16349 server_base.cc:1034] running on GCE node
I20260927 10:50:37.429863 16349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:37.430514 16349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:37.431685 16349 hybrid_clock.cc:648] HybridClock initialized: now 1790506237431663 us; error 30 us; skew 500 ppm
I20260927 10:50:37.433702 16349 webserver.cc:460] Webserver started at http://127.6.31.131:45007/ using document root <none> and password file <none>
I20260927 10:50:37.434306 16349 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:37.434370 16349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:37.434582 16349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:37.436810 16349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "eb04563e595843de8b8e38f2933176fa"
format_stamp: "Formatted at 2026-09-27 10:50:37 on dist-test-slave-c60x"
server_key: "c12ab11060b3dff5f55c7f4b7aa182f8"
server_key_iv: "3629f681939761be8003dadcdf86bc3c"
server_key_version: "encryptionkey@0"
I20260927 10:50:37.437402 16349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "eb04563e595843de8b8e38f2933176fa"
format_stamp: "Formatted at 2026-09-27 10:50:37 on dist-test-slave-c60x"
server_key: "c12ab11060b3dff5f55c7f4b7aa182f8"
server_key_iv: "3629f681939761be8003dadcdf86bc3c"
server_key_version: "encryptionkey@0"
I20260927 10:50:37.441354 16349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:50:37.444041 16363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:37.445489 16349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:50:37.445642 16349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "eb04563e595843de8b8e38f2933176fa"
format_stamp: "Formatted at 2026-09-27 10:50:37 on dist-test-slave-c60x"
server_key: "c12ab11060b3dff5f55c7f4b7aa182f8"
server_key_iv: "3629f681939761be8003dadcdf86bc3c"
server_key_version: "encryptionkey@0"
I20260927 10:50:37.445782 16349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:37.450580 16349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:37.451270 16349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:37.451458 16349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:37.452013 16349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:37.452942 16349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:37.453022 16349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:37.453092 16349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:37.453142 16349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:37.462042 16349 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:39967
I20260927 10:50:37.462100 16476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:39967 every 8 connection(s)
I20260927 10:50:37.462983 16349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:50:37.468909 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:37.464907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:57001 (local address 127.6.31.190:37143)
0927 10:50:37.465058 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:37.465061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:37.466112 (+ 1051us) server_negotiation.cc:407] Connection header received
0927 10:50:37.467437 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:37.467439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:37.467477 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:37.467552 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:37.468550 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:37.468553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:37.468555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:50:37.468566 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:37.468582 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:50:37.468590 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:37.468670 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:37.468765 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 10:50:37.468803 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:50:37.469525 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 16349
I20260927 10:50:37.469648 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:50:37.469815 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:37.465324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37143 (local address 127.6.31.131:57001)
0927 10:50:37.465969 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:37.466007 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:37.467242 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:37.467701 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:37.467707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:37.467792 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:37.468311 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:50:37.468313 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:37.468352 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:50:37.468415 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:37.468655 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:37.468659 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:37.468663 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:50:37.468781 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 10:50:37.468983 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260927 10:50:37.469959 6270 external_mini_cluster.cc:1428] Setting key eb009b3a4a99f5dfdf765561508ba8d2
I20260927 10:50:37.471062 16477 heartbeater.cc:347] Connected to a master server at 127.6.31.190:37143
I20260927 10:50:37.471329 16477 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:37.471784 16477 heartbeater.cc:511] Master 127.6.31.190:37143 requested a full tablet report, sending...
I20260927 10:50:37.472815 16026 ts_manager.cc:194] Registered new tserver with Master: eb04563e595843de8b8e38f2933176fa (127.6.31.131:39967)
I20260927 10:50:37.484808 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:50:37.487790 16066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:50:37.486760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.6.31.190:37143)
0927 10:50:37.486906 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:37.486913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:37.487129 (+ 216us) server_negotiation.cc:407] Connection header received
0927 10:50:37.487296 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:37.487305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:37.487338 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:37.487422 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:37.487664 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57648: BlockingRecv error: recv got EOF from 127.0.0.1:57648 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:50:37.488142 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 16087
I20260927 10:50:37.494112 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 16218
I20260927 10:50:37.499261 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 16349
I20260927 10:50:37.504266 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 15996
2026-09-27T10:50:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (832 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:50:37 dist-test-slave-c60x krb5kdc[16487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:50:37 dist-test-slave-c60x krb5kdc[16487](info): set up 2 sockets
Sep 27 10:50:37 dist-test-slave-c60x krb5kdc[16487](info): commencing operation
krb5kdc: starting...
W20260927 10:50:38.552213 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:50:38 dist-test-slave-c60x krb5kdc[16487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:50:38.679853 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46045
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:38.791525 16500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:38.791810 16500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:38.791879 16500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:38.796270 16500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:50:38.796335 16500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:38.796355 16500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:38.796373 16500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:50:38.796425 16500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:50:38.801864 16500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:46045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:38.802997 16500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:38.804069 16500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:38.812980 16508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:38.813045 16506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:38.813040 16505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:38.813318 16500 server_base.cc:1034] running on GCE node
I20260927 10:50:38.813881 16500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:38.815018 16500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:38.816272 16500 hybrid_clock.cc:648] HybridClock initialized: now 1790506238816241 us; error 49 us; skew 500 ppm
Sep 27 10:50:38 dist-test-slave-c60x krb5kdc[16487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506238, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:38.825947 16500 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:50:38.827062 16500 webserver.cc:460] Webserver started at http://127.6.31.190:43377/ using document root <none> and password file <none>
I20260927 10:50:38.827710 16500 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:38.827801 16500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:38.828035 16500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:38.830215 16500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "13737be44f1348daa2dd6dbc9f5fcd21"
format_stamp: "Formatted at 2026-09-27 10:50:38 on dist-test-slave-c60x"
server_key: "75cc73c4817b8022337f5c7d5db9e85f"
server_key_iv: "cd0295007acf26d06eb9ac4210fe8719"
server_key_version: "encryptionkey@0"
I20260927 10:50:38.830804 16500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "13737be44f1348daa2dd6dbc9f5fcd21"
format_stamp: "Formatted at 2026-09-27 10:50:38 on dist-test-slave-c60x"
server_key: "75cc73c4817b8022337f5c7d5db9e85f"
server_key_iv: "cd0295007acf26d06eb9ac4210fe8719"
server_key_version: "encryptionkey@0"
I20260927 10:50:38.834671 16500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 10:50:38.837275 16515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:38.838584 16500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:50:38.838745 16500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "13737be44f1348daa2dd6dbc9f5fcd21"
format_stamp: "Formatted at 2026-09-27 10:50:38 on dist-test-slave-c60x"
server_key: "75cc73c4817b8022337f5c7d5db9e85f"
server_key_iv: "cd0295007acf26d06eb9ac4210fe8719"
server_key_version: "encryptionkey@0"
I20260927 10:50:38.838869 16500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:38.852981 16500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:38.856451 16500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:38.856664 16500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:38.864162 16500 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:46045
I20260927 10:50:38.864467 16567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:46045 every 8 connection(s)
I20260927 10:50:38.865222 16500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:50:38.866626 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 16500
I20260927 10:50:38.866763 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:50:38.867057 6270 external_mini_cluster.cc:1428] Setting key 5fe659eeab51aa08195576577793c275
I20260927 10:50:38.868582 16568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:38.875464 16568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21: Bootstrap starting.
Sep 27 10:50:38 dist-test-slave-c60x krb5kdc[16487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:50:38.878237 16568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:38.879186 16568 log.cc:824] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:38.881633 16568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21: No bootstrap required, opened a new log
I20260927 10:50:38.882427 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:38.868441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.6.31.190:46045)
0927 10:50:38.868872 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:38.868890 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:38.868930 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:50:38.869744 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:38.869766 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:38.870165 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:38.870510 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:38.871286 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:38.872787 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:38.873644 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:38.874340 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:38.874511 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:38.877301 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:38.877307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:38.877323 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:38.877358 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:38.879410 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:38.879913 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:38.879918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:38.879922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:38.880062 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:38.880332 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:38.880335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:38.880336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:38.880831 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:38.881029 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:38.881223 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 10:50:38.881511 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":142,"threads_started":1}
I20260927 10:50:38.885602 16568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13737be44f1348daa2dd6dbc9f5fcd21" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46045 } }
I20260927 10:50:38.885818 16568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:38.885854 16568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13737be44f1348daa2dd6dbc9f5fcd21, State: Initialized, Role: FOLLOWER
I20260927 10:50:38.886260 16568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13737be44f1348daa2dd6dbc9f5fcd21" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46045 } }
I20260927 10:50:38.886377 16568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:50:38.886422 16568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:50:38.886492 16568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:38.887449 16568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13737be44f1348daa2dd6dbc9f5fcd21" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46045 } }
I20260927 10:50:38.887734 16568 leader_election.cc:302] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13737be44f1348daa2dd6dbc9f5fcd21; no voters:
I20260927 10:50:38.887984 16568 leader_election.cc:288] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:50:38.888159 16573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:38.888445 16573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [term 1 LEADER]: Becoming Leader. State: Replica: 13737be44f1348daa2dd6dbc9f5fcd21, State: Running, Role: LEADER
I20260927 10:50:38.888724 16573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13737be44f1348daa2dd6dbc9f5fcd21" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46045 } }
I20260927 10:50:38.889307 16568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:50:38.890497 16575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13737be44f1348daa2dd6dbc9f5fcd21. Latest consensus state: current_term: 1 leader_uuid: "13737be44f1348daa2dd6dbc9f5fcd21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13737be44f1348daa2dd6dbc9f5fcd21" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46045 } } }
I20260927 10:50:38.890640 16575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:38.890931 16574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13737be44f1348daa2dd6dbc9f5fcd21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13737be44f1348daa2dd6dbc9f5fcd21" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46045 } } }
I20260927 10:50:38.891019 16574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21 [sys.catalog]: This master's current role is: LEADER
I20260927 10:50:38.891160 16582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:50:38.893781 16582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:50:38.899638 16582 catalog_manager.cc:1347] Generated new cluster ID: dbf67eace0434efbb0828efd7caee92b
I20260927 10:50:38.899716 16582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:50:38.920866 16582 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:50:38.922139 16582 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:50:38.931651 16582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21: Generated new TSK 0
I20260927 10:50:38.932441 16582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:50:38.966820 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46045
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:39.078428 16594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:39.078694 16594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:39.078755 16594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:39.082811 16594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:39.082868 16594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:39.082927 16594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:50:39.088063 16594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:39.089011 16594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:39.090066 16594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:39.099766 16599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:39.099763 16602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:39.099922 16594 server_base.cc:1034] running on GCE node
W20260927 10:50:39.100076 16600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:39.100468 16594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:39.101074 16594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:39.102231 16594 hybrid_clock.cc:648] HybridClock initialized: now 1790506239102218 us; error 26 us; skew 500 ppm
Sep 27 10:50:39 dist-test-slave-c60x krb5kdc[16487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:39.104997 16594 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:50:39.105906 16594 webserver.cc:460] Webserver started at http://127.6.31.129:42181/ using document root <none> and password file <none>
I20260927 10:50:39.106498 16594 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:39.106590 16594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:39.106827 16594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:39.108990 16594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "350741731e4447c99ef7a1060f5daa3e"
format_stamp: "Formatted at 2026-09-27 10:50:39 on dist-test-slave-c60x"
server_key: "4eb3b2beb6774a29c30e0671377bed83"
server_key_iv: "fae4e3c28c81c32bc9acc4a391645d83"
server_key_version: "encryptionkey@0"
I20260927 10:50:39.109551 16594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "350741731e4447c99ef7a1060f5daa3e"
format_stamp: "Formatted at 2026-09-27 10:50:39 on dist-test-slave-c60x"
server_key: "4eb3b2beb6774a29c30e0671377bed83"
server_key_iv: "fae4e3c28c81c32bc9acc4a391645d83"
server_key_version: "encryptionkey@0"
I20260927 10:50:39.113225 16594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:50:39.115697 16609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:39.116899 16594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:39.117028 16594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "350741731e4447c99ef7a1060f5daa3e"
format_stamp: "Formatted at 2026-09-27 10:50:39 on dist-test-slave-c60x"
server_key: "4eb3b2beb6774a29c30e0671377bed83"
server_key_iv: "fae4e3c28c81c32bc9acc4a391645d83"
server_key_version: "encryptionkey@0"
I20260927 10:50:39.117137 16594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:39.152029 16594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:39.155419 16594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:39.155627 16594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:39.156243 16594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:39.157074 16594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:39.157146 16594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:39.157219 16594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:39.157262 16594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:39.166088 16594 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:34203
I20260927 10:50:39.166124 16722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:34203 every 8 connection(s)
I20260927 10:50:39.166988 16594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:50:39.173261 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 16594
I20260927 10:50:39.173403 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:50:39.173676 6270 external_mini_cluster.cc:1428] Setting key 649998949c5d6003e9242c5b1d51c7a9
Sep 27 10:50:39 dist-test-slave-c60x krb5kdc[16487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:50:39.181095 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.168965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:43609 (local address 127.6.31.190:46045)
0927 10:50:39.169103 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.169107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.169841 (+ 734us) server_negotiation.cc:407] Connection header received
0927 10:50:39.171212 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.171217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.171274 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.171391 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:39.172548 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.173095 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.174408 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.174648 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.174764 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.177315 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:39.177319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:39.177321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:39.177352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:39.178861 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:39.179466 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:39.179472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:39.179479 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:39.179557 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:39.179915 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:39.179919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:39.179920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:39.180121 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:39.180228 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.180783 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.180924 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:50:39.182026 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.169232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46045 (local address 127.6.31.129:43609)
0927 10:50:39.169699 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:39.169739 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:39.170975 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:39.171554 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:39.171560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:39.171915 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:39.172391 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.172404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.173222 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.173226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.174249 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.174258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.174801 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.174805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.174985 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.175661 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:39.175677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:39.177138 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:39.178986 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:39.178992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:39.179010 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:39.179355 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:39.179654 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:39.179658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:39.179660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:39.179806 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:39.180249 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:39.180256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:39.180538 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 10:50:39.180728 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 10:50:39.181109 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:50:39.183437 16723 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46045
I20260927 10:50:39.183820 16723 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:39.184368 16723 heartbeater.cc:511] Master 127.6.31.190:46045 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:50:39.186060 16532 ts_manager.cc:194] Registered new tserver with Master: 350741731e4447c99ef7a1060f5daa3e (127.6.31.129:34203)
I20260927 10:50:39.187686 16532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:43609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:50:39.203236 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46045
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:39.313831 16728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:39.314119 16728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:39.314165 16728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:39.319005 16728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:39.319128 16728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:39.319209 16728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:50:39.324720 16728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:39.325758 16728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:39.326741 16728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:39.336189 16733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:39.336194 16736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:39.336175 16734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:39.336462 16728 server_base.cc:1034] running on GCE node
I20260927 10:50:39.336921 16728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:39.337577 16728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:39.338788 16728 hybrid_clock.cc:648] HybridClock initialized: now 1790506239338769 us; error 25 us; skew 500 ppm
Sep 27 10:50:39 dist-test-slave-c60x krb5kdc[16487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:39.341568 16728 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:50:39.342427 16728 webserver.cc:460] Webserver started at http://127.6.31.130:39075/ using document root <none> and password file <none>
I20260927 10:50:39.342980 16728 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:39.343036 16728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:39.343302 16728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:39.345315 16728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "fc87202f001c4aec83cae53d4046ad62"
format_stamp: "Formatted at 2026-09-27 10:50:39 on dist-test-slave-c60x"
server_key: "a18454091aa6ca191d59d976f23a870c"
server_key_iv: "3041adb697d940c3cd733cea4e365862"
server_key_version: "encryptionkey@0"
I20260927 10:50:39.345804 16728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "fc87202f001c4aec83cae53d4046ad62"
format_stamp: "Formatted at 2026-09-27 10:50:39 on dist-test-slave-c60x"
server_key: "a18454091aa6ca191d59d976f23a870c"
server_key_iv: "3041adb697d940c3cd733cea4e365862"
server_key_version: "encryptionkey@0"
I20260927 10:50:39.349341 16728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:50:39.351629 16743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:39.352836 16728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:50:39.352967 16728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "fc87202f001c4aec83cae53d4046ad62"
format_stamp: "Formatted at 2026-09-27 10:50:39 on dist-test-slave-c60x"
server_key: "a18454091aa6ca191d59d976f23a870c"
server_key_iv: "3041adb697d940c3cd733cea4e365862"
server_key_version: "encryptionkey@0"
I20260927 10:50:39.353075 16728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:39.390158 16728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:39.393533 16728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:39.393747 16728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:39.394357 16728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:39.395231 16728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:39.395296 16728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:39.395363 16728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:39.395409 16728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:39.404261 16728 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:44563
I20260927 10:50:39.404284 16856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:44563 every 8 connection(s)
I20260927 10:50:39.405121 16728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:50:39.410115 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 16728
I20260927 10:50:39.410243 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:50:39.410503 6270 external_mini_cluster.cc:1428] Setting key 8bae7e23308ce0333773f35cd810ad26
Sep 27 10:50:39 dist-test-slave-c60x krb5kdc[16487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:50:39.420635 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.406882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:54353 (local address 127.6.31.190:46045)
0927 10:50:39.407033 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.407038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.408025 (+ 987us) server_negotiation.cc:407] Connection header received
0927 10:50:39.409573 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.409596 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.409649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.409751 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:39.411316 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.411934 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.413226 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.413640 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.413743 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.416764 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:39.416769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:39.416772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:39.416807 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:39.418290 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:39.418834 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:39.418842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:39.418843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:39.418917 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:39.419341 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:39.419344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:39.419346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:39.419547 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:39.419664 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.420327 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.420460 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:50:39.421761 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.407393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46045 (local address 127.6.31.130:54353)
0927 10:50:39.407865 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:39.407908 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:39.409216 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:39.409968 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:39.409976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:39.410531 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:39.411166 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.411183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.412092 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.412095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.413074 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.413084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.413834 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.413840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.414074 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.414782 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:39.414810 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:39.416576 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:39.418418 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:39.418424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:39.418442 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:39.418710 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:39.419009 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:39.419014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:39.419016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:39.419238 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:39.419662 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:39.419669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:39.419957 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 10:50:39.420265 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 10:50:39.420607 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
I20260927 10:50:39.423134 16857 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46045
I20260927 10:50:39.423440 16857 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:39.424104 16857 heartbeater.cc:511] Master 127.6.31.190:46045 requested a full tablet report, sending...
I20260927 10:50:39.425400 16532 ts_manager.cc:194] Registered new tserver with Master: fc87202f001c4aec83cae53d4046ad62 (127.6.31.130:44563)
I20260927 10:50:39.426208 16532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:54353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:50:39.441651 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46045
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:50:39.553395 16862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:50:39.553642 16862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:50:39.553714 16862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:50:39.557817 16862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:50:39.557876 16862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:50:39.557966 16862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:50:39.563133 16862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:50:39.564144 16862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:50:39.565119 16862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:50:39.574409 16870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:39.574442 16862 server_base.cc:1034] running on GCE node
W20260927 10:50:39.574363 16867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:50:39.574366 16868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:50:39.575201 16862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:50:39.575781 16862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:50:39.576941 16862 hybrid_clock.cc:648] HybridClock initialized: now 1790506239576927 us; error 25 us; skew 500 ppm
Sep 27 10:50:39 dist-test-slave-c60x krb5kdc[16487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:50:39.579718 16862 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:50:39.580622 16862 webserver.cc:460] Webserver started at http://127.6.31.131:44243/ using document root <none> and password file <none>
I20260927 10:50:39.581168 16862 fs_manager.cc:359] Metadata directory not provided
I20260927 10:50:39.581264 16862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:50:39.581486 16862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:50:39.583601 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5"
format_stamp: "Formatted at 2026-09-27 10:50:39 on dist-test-slave-c60x"
server_key: "3b9ab08cac51625c401eca42b1e820bc"
server_key_iv: "e4e40228e962e9c4a233a748fc6b3f8d"
server_key_version: "encryptionkey@0"
I20260927 10:50:39.584156 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5"
format_stamp: "Formatted at 2026-09-27 10:50:39 on dist-test-slave-c60x"
server_key: "3b9ab08cac51625c401eca42b1e820bc"
server_key_iv: "e4e40228e962e9c4a233a748fc6b3f8d"
server_key_version: "encryptionkey@0"
I20260927 10:50:39.587747 16862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:50:39.590209 16877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:39.591634 16862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:50:39.591784 16862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5"
format_stamp: "Formatted at 2026-09-27 10:50:39 on dist-test-slave-c60x"
server_key: "3b9ab08cac51625c401eca42b1e820bc"
server_key_iv: "e4e40228e962e9c4a233a748fc6b3f8d"
server_key_version: "encryptionkey@0"
I20260927 10:50:39.591897 16862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:50:39.608198 16862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:50:39.611374 16862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:50:39.611562 16862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:50:39.612133 16862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:50:39.612931 16862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:50:39.613001 16862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:39.613071 16862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:50:39.613124 16862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:50:39.622068 16862 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:37097
I20260927 10:50:39.622087 16990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:37097 every 8 connection(s)
I20260927 10:50:39.622946 16862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:50:39.628497 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 16862
I20260927 10:50:39.628635 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:50:39.628960 6270 external_mini_cluster.cc:1428] Setting key 11b09aa6867b48766a34e0689bc20a96
Sep 27 10:50:39 dist-test-slave-c60x krb5kdc[16487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:50:39.637526 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.624791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:58361 (local address 127.6.31.190:46045)
0927 10:50:39.624930 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.624934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.625734 (+ 800us) server_negotiation.cc:407] Connection header received
0927 10:50:39.627073 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.627077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.627160 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.627347 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:39.628985 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.629711 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.631299 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.631524 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.631639 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.634239 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:39.634243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:39.634246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:39.634270 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:39.635828 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:39.636296 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:39.636299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:39.636301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:39.636370 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:39.636649 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:39.636651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:39.636652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:39.636830 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:39.636902 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.637321 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.637410 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:50:39.638372 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.625090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46045 (local address 127.6.31.131:58361)
0927 10:50:39.625569 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:39.625610 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:39.626860 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:39.627508 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:39.627515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:39.627919 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:50:39.628705 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.628725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.629843 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.629846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.631169 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.631185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.631684 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.631688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.631856 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.632485 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:50:39.632501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:50:39.634078 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:50:39.635937 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:39.635942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:39.635953 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:39.636213 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:39.636448 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:50:39.636450 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:50:39.636452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:50:39.636574 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:50:39.636911 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:50:39.636916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:50:39.637127 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 10:50:39.637277 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 10:50:39.637537 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":109,"threads_started":1}
I20260927 10:50:39.639539 16991 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46045
I20260927 10:50:39.639765 16991 heartbeater.cc:464] Registering TS with master...
I20260927 10:50:39.640237 16991 heartbeater.cc:511] Master 127.6.31.190:46045 requested a full tablet report, sending...
I20260927 10:50:39.641263 16532 ts_manager.cc:194] Registered new tserver with Master: fe6a2f04228f4e2290251d8cdb3cc9f5 (127.6.31.131:37097)
I20260927 10:50:39.642040 16532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:58361
I20260927 10:50:39.645356 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:50:39.654290 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.647311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.6.31.190:46045)
0927 10:50:39.647439 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.647442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.647643 (+ 201us) server_negotiation.cc:407] Connection header received
0927 10:50:39.647727 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.647730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.647772 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.647837 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:50:39.648481 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.649011 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.649873 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.650071 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.650141 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.651558 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:50:39.651561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:50:39.651563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:50:39.651584 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:50:39.652968 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:39.653347 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:39.653350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:39.653351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:39.653417 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:50:39.653634 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:50:39.653636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:50:39.653637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:50:39.653832 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:50:39.653928 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.654094 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.654176 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:50:39.657114 16532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:50:39.660131 16532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:50:39.674811 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.668757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.6.31.129:34203)
0927 10:50:39.669023 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.669037 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.669571 (+ 534us) server_negotiation.cc:407] Connection header received
0927 10:50:39.669641 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.669645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.669798 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.670013 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:39.670477 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.671646 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.673065 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.673943 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.674070 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.674236 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.674534 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.674644 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260927 10:50:39.674865 17002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.668034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:37097 (local address 127.0.0.1:45656)
0927 10:50:39.668678 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:39.668731 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:39.668861 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:39.669977 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:39.669987 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:39.670013 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:39.670159 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.670167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.671669 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.671672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.673106 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.673117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.674114 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.674118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.674299 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.674353 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 10:50:39.674484 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 10:50:39.674647 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":180,"thread_start_us":78,"threads_started":1}
I20260927 10:50:39.674865 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.668175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:34203 (local address 127.0.0.1:53370)
0927 10:50:39.669090 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:39.669122 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:39.669220 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:39.670189 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:39.670191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:39.670205 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:39.670346 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.670354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.671774 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.671779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.672944 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.672953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.674114 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.674118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.674355 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.674372 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:50:39.674484 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:50:39.674647 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":155,"thread_start_us":122,"threads_started":1}
I20260927 10:50:39.675776 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.668720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:44563 (local address 127.0.0.1:54782)
0927 10:50:39.669258 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:39.669272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:39.669355 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:39.670218 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:39.670224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:39.670247 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:39.670355 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.670360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.671403 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.671409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.673105 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.673117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.675366 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.675373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.675494 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.675522 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:50:39.675598 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:50:39.675674 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":173,"thread_start_us":107,"threads_started":1}
I20260927 10:50:39.675904 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.668181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.6.31.131:37097)
0927 10:50:39.669475 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.669487 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.669511 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:50:39.669594 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.669598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.669798 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.669971 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:39.670298 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.671526 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.673233 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.673988 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.674086 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.674228 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.675058 (+ 830us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.675751 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":115,"threads_started":1}
I20260927 10:50:39.676290 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.669145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.6.31.130:44563)
0927 10:50:39.669548 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.669560 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.669584 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:50:39.669660 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.669668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.669869 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.670050 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:39.670473 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.671273 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.674294 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.675235 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.675362 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.675504 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.676015 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.676120 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":90,"threads_started":1}
I20260927 10:50:39.678375 16925 tablet_service.cc:1467] Processing CreateTablet for tablet 55264b81786945b1918bd8b830f6b81e (DEFAULT_TABLE table=test-table [id=52679a1c0b13423da36a13e33cf52137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:39.678459 16657 tablet_service.cc:1467] Processing CreateTablet for tablet 55264b81786945b1918bd8b830f6b81e (DEFAULT_TABLE table=test-table [id=52679a1c0b13423da36a13e33cf52137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:39.678694 16791 tablet_service.cc:1467] Processing CreateTablet for tablet 55264b81786945b1918bd8b830f6b81e (DEFAULT_TABLE table=test-table [id=52679a1c0b13423da36a13e33cf52137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:50:39.679493 16925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55264b81786945b1918bd8b830f6b81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:39.679505 16657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55264b81786945b1918bd8b830f6b81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:39.679514 16791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55264b81786945b1918bd8b830f6b81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:50:39.685400 17008 tablet_bootstrap.cc:492] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: Bootstrap starting.
I20260927 10:50:39.687001 17010 tablet_bootstrap.cc:492] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: Bootstrap starting.
I20260927 10:50:39.688117 17008 tablet_bootstrap.cc:654] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:39.688865 17008 log.cc:824] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:39.689193 17010 tablet_bootstrap.cc:654] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:39.689565 17009 tablet_bootstrap.cc:492] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e: Bootstrap starting.
I20260927 10:50:39.689951 17010 log.cc:824] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:39.691627 17009 tablet_bootstrap.cc:654] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e: Neither blocks nor log segments found. Creating new log.
I20260927 10:50:39.692075 17010 tablet_bootstrap.cc:492] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: No bootstrap required, opened a new log
I20260927 10:50:39.692273 17010 ts_tablet_manager.cc:1397] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:50:39.692400 17009 log.cc:824] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:50:39.692559 17008 tablet_bootstrap.cc:492] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: No bootstrap required, opened a new log
I20260927 10:50:39.692735 17008 ts_tablet_manager.cc:1397] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:50:39.694249 17009 tablet_bootstrap.cc:492] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e: No bootstrap required, opened a new log
I20260927 10:50:39.694420 17009 ts_tablet_manager.cc:1397] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:50:39.695705 17010 raft_consensus.cc:348] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } }
I20260927 10:50:39.695978 17010 raft_consensus.cc:374] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:39.696033 17010 raft_consensus.cc:729] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc87202f001c4aec83cae53d4046ad62, State: Initialized, Role: FOLLOWER
I20260927 10:50:39.696518 17010 consensus_queue.cc:260] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } }
I20260927 10:50:39.697147 16857 heartbeater.cc:503] Master 127.6.31.190:46045 was elected leader, sending a full tablet report...
I20260927 10:50:39.697513 17010 ts_tablet_manager.cc:1428] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:50:39.696861 17008 raft_consensus.cc:348] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } }
I20260927 10:50:39.698190 17008 raft_consensus.cc:374] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:39.698257 17008 raft_consensus.cc:729] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6a2f04228f4e2290251d8cdb3cc9f5, State: Initialized, Role: FOLLOWER
I20260927 10:50:39.698704 17008 consensus_queue.cc:260] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } }
I20260927 10:50:39.698663 17009 raft_consensus.cc:348] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } }
I20260927 10:50:39.698905 17009 raft_consensus.cc:374] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:50:39.698971 17009 raft_consensus.cc:729] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350741731e4447c99ef7a1060f5daa3e, State: Initialized, Role: FOLLOWER
I20260927 10:50:39.699494 16991 heartbeater.cc:503] Master 127.6.31.190:46045 was elected leader, sending a full tablet report...
I20260927 10:50:39.699440 17009 consensus_queue.cc:260] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } }
I20260927 10:50:39.699723 17008 ts_tablet_manager.cc:1428] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:50:39.700284 17009 ts_tablet_manager.cc:1428] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:50:39.700572 16723 heartbeater.cc:503] Master 127.6.31.190:46045 was elected leader, sending a full tablet report...
I20260927 10:50:39.855345 17016 raft_consensus.cc:482] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:50:39.855643 17016 raft_consensus.cc:504] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } }
I20260927 10:50:39.856904 17016 leader_election.cc:288] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe6a2f04228f4e2290251d8cdb3cc9f5 (127.6.31.131:37097), fc87202f001c4aec83cae53d4046ad62 (127.6.31.130:44563)
I20260927 10:50:39.860893 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.857243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:56859 (local address 127.6.31.130:44563)
0927 10:50:39.857393 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.857398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.857711 (+ 313us) server_negotiation.cc:407] Connection header received
0927 10:50:39.857866 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.857869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.857926 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.858012 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:39.858295 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.858811 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.859964 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.860349 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.860491 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.860545 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.860708 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.860771 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 10:50:39.860924 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.857169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:44563 (local address 127.6.31.129:56859)
0927 10:50:39.857626 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:39.857638 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:39.857751 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:39.858026 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:39.858028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:39.858063 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:39.858192 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.858197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.858920 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.858924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.859848 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.859856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.860479 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.860483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.860579 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.860601 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:50:39.860695 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:50:39.860762 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":58,"spinlock_wait_cycles":1280,"thread_start_us":121,"threads_started":1}
I20260927 10:50:39.861001 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.857242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:37991 (local address 127.6.31.131:37097)
0927 10:50:39.857393 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.857398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.857659 (+ 261us) server_negotiation.cc:407] Connection header received
0927 10:50:39.857867 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.857870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.857927 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.858010 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:50:39.858309 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.858807 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.860049 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.860450 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.860579 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.860629 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.860804 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.860868 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:50:39.861234 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.857144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:37097 (local address 127.6.31.129:37991)
0927 10:50:39.857565 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:39.857577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:50:39.857732 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:50:39.858022 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:50:39.858025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:50:39.858057 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:50:39.858200 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.858205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.858921 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.858926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.859927 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:50:39.859936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.860571 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.860575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:50:39.860677 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.860704 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:50:39.860789 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:50:39.860863 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":22,"spinlock_wait_cycles":12544,"thread_start_us":129,"threads_started":1}
I20260927 10:50:39.861732 16811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55264b81786945b1918bd8b830f6b81e" candidate_uuid: "350741731e4447c99ef7a1060f5daa3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc87202f001c4aec83cae53d4046ad62" is_pre_election: true
I20260927 10:50:39.861838 16945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55264b81786945b1918bd8b830f6b81e" candidate_uuid: "350741731e4447c99ef7a1060f5daa3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" is_pre_election: true
I20260927 10:50:39.862064 16811 raft_consensus.cc:2454] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 350741731e4447c99ef7a1060f5daa3e in term 0.
I20260927 10:50:39.862181 16945 raft_consensus.cc:2454] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 350741731e4447c99ef7a1060f5daa3e in term 0.
I20260927 10:50:39.862561 16613 leader_election.cc:302] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 350741731e4447c99ef7a1060f5daa3e, fc87202f001c4aec83cae53d4046ad62, fe6a2f04228f4e2290251d8cdb3cc9f5; no voters:
I20260927 10:50:39.862813 17016 raft_consensus.cc:2781] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:50:39.862890 17016 raft_consensus.cc:482] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:50:39.862939 17016 raft_consensus.cc:3037] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:39.863996 17016 raft_consensus.cc:504] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } }
I20260927 10:50:39.864400 17016 leader_election.cc:288] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [CANDIDATE]: Term 1 election: Requested vote from peers fe6a2f04228f4e2290251d8cdb3cc9f5 (127.6.31.131:37097), fc87202f001c4aec83cae53d4046ad62 (127.6.31.130:44563)
I20260927 10:50:39.864776 16811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55264b81786945b1918bd8b830f6b81e" candidate_uuid: "350741731e4447c99ef7a1060f5daa3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc87202f001c4aec83cae53d4046ad62"
I20260927 10:50:39.864882 16811 raft_consensus.cc:3037] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:39.865895 16811 raft_consensus.cc:2454] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 350741731e4447c99ef7a1060f5daa3e in term 1.
I20260927 10:50:39.866163 16613 leader_election.cc:302] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 350741731e4447c99ef7a1060f5daa3e, fc87202f001c4aec83cae53d4046ad62; no voters:
I20260927 10:50:39.866348 17016 raft_consensus.cc:2781] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:50:39.866604 17016 raft_consensus.cc:686] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 1 LEADER]: Becoming Leader. State: Replica: 350741731e4447c99ef7a1060f5daa3e, State: Running, Role: LEADER
I20260927 10:50:39.866948 17016 consensus_queue.cc:237] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } }
I20260927 10:50:39.871408 16945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55264b81786945b1918bd8b830f6b81e" candidate_uuid: "350741731e4447c99ef7a1060f5daa3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5"
I20260927 10:50:39.871567 16945 raft_consensus.cc:3037] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:50:39.871816 16531 catalog_manager.cc:5495] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 350741731e4447c99ef7a1060f5daa3e (127.6.31.129). New cstate: current_term: 1 leader_uuid: "350741731e4447c99ef7a1060f5daa3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "350741731e4447c99ef7a1060f5daa3e" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 } health_report { overall_health: UNKNOWN } } }
I20260927 10:50:39.872956 16945 raft_consensus.cc:2454] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 350741731e4447c99ef7a1060f5daa3e in term 1.
W20260927 10:50:39.873903 16992 tablet.cc:2369] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:50:39.906250 16858 tablet.cc:2369] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:50:39.916899 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.913424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.6.31.129:34203)
0927 10:50:39.913559 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:50:39.913562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:50:39.913574 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:50:39.913646 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:50:39.913648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:50:39.913703 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:50:39.913816 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:50:39.914051 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.914584 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.915519 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:50:39.915749 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:50:39.915844 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:50:39.916104 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:50:39.916543 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:50:39.916635 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:50:39.916722 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 10:50:39.916772 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 10:50:39.918318 16724 tablet.cc:2369] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:50:39.922904 16811 raft_consensus.cc:1261] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [term 1 FOLLOWER]: Refusing update from remote peer 350741731e4447c99ef7a1060f5daa3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:39.922907 16945 raft_consensus.cc:1261] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [term 1 FOLLOWER]: Refusing update from remote peer 350741731e4447c99ef7a1060f5daa3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:50:39.923632 17016 consensus_queue.cc:1035] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc87202f001c4aec83cae53d4046ad62" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:39.923839 17019 consensus_queue.cc:1035] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6a2f04228f4e2290251d8cdb3cc9f5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:50:39.942241 16531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:39.942432 16531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:50:39.945179 16531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13737be44f1348daa2dd6dbc9f5fcd21: Sending DeleteTablet for 3 replicas of tablet 55264b81786945b1918bd8b830f6b81e
I20260927 10:50:39.946128 16925 tablet_service.cc:1514] Processing DeleteTablet for tablet 55264b81786945b1918bd8b830f6b81e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:39 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:45656
I20260927 10:50:39.946125 16657 tablet_service.cc:1514] Processing DeleteTablet for tablet 55264b81786945b1918bd8b830f6b81e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:39 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:53370
I20260927 10:50:39.946223 16791 tablet_service.cc:1514] Processing DeleteTablet for tablet 55264b81786945b1918bd8b830f6b81e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:50:39 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:54782
I20260927 10:50:39.946736 17033 tablet_replica.cc:330] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e: stopping tablet replica
I20260927 10:50:39.947116 17033 raft_consensus.cc:2229] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:50:39.947535 17033 raft_consensus.cc:2258] T 55264b81786945b1918bd8b830f6b81e P 350741731e4447c99ef7a1060f5daa3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:39.947687 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 16594
I20260927 10:50:39.948321 17034 tablet_replica.cc:330] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: stopping tablet replica
I20260927 10:50:39.948637 17034 raft_consensus.cc:2229] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:39.948972 17034 raft_consensus.cc:2258] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:39.949349 17032 tablet_replica.cc:330] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: stopping tablet replica
I20260927 10:50:39.949685 17032 raft_consensus.cc:2229] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:50:39.949965 17032 raft_consensus.cc:2258] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:50:39.950603 17034 ts_tablet_manager.cc:1905] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:39.952930 17034 ts_tablet_manager.cc:1918] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:39.953054 17034 log.cc:1196] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/55264b81786945b1918bd8b830f6b81e
I20260927 10:50:39.953351 17034 ts_tablet_manager.cc:1939] T 55264b81786945b1918bd8b830f6b81e P fc87202f001c4aec83cae53d4046ad62: Deleting consensus metadata
I20260927 10:50:39.954578 16519 catalog_manager.cc:4841] TS fc87202f001c4aec83cae53d4046ad62 (127.6.31.130:44563): tablet 55264b81786945b1918bd8b830f6b81e (table test-table [id=52679a1c0b13423da36a13e33cf52137]) successfully deleted
W20260927 10:50:39.955348 16516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:34203 (error 108)
I20260927 10:50:39.955610 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 16728
I20260927 10:50:39.955659 17032 ts_tablet_manager.cc:1905] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:50:39.956612 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:50:39.956309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:34203 (local address 127.0.0.1:53402)
0927 10:50:39.956436 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 10:50:39.956509 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:34203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:50:39.957231 16516 catalog_manager.cc:4584] TS 350741731e4447c99ef7a1060f5daa3e (127.6.31.129:34203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55264b81786945b1918bd8b830f6b81e: Network error: Client connection negotiation failed: client connection to 127.6.31.129:34203: connect: Connection refused (error 111)
I20260927 10:50:39.958691 17032 ts_tablet_manager.cc:1918] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:50:39.958776 17032 log.cc:1196] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/55264b81786945b1918bd8b830f6b81e
I20260927 10:50:39.959129 17032 ts_tablet_manager.cc:1939] T 55264b81786945b1918bd8b830f6b81e P fe6a2f04228f4e2290251d8cdb3cc9f5: Deleting consensus metadata
I20260927 10:50:39.960273 16517 catalog_manager.cc:4841] TS fe6a2f04228f4e2290251d8cdb3cc9f5 (127.6.31.131:37097): tablet 55264b81786945b1918bd8b830f6b81e (table test-table [id=52679a1c0b13423da36a13e33cf52137]) successfully deleted
I20260927 10:50:39.962222 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 16862
I20260927 10:50:39.967703 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 16500
2026-09-27T10:50:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2464 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:50:40 dist-test-slave-c60x krb5kdc[17038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:50:40 dist-test-slave-c60x krb5kdc[17038](info): set up 2 sockets
Sep 27 10:50:40 dist-test-slave-c60x krb5kdc[17038](info): commencing operation
krb5kdc: starting...
W20260927 10:50:41.016686 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:50:41 dist-test-slave-c60x krb5kdc[17038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:50:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.31.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.31.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/rangeradmin_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/rangeradmin_127.6.31.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.31.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.31.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/rangerlookup_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/rangerlookup_127.6.31.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.31.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.31.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/HTTP_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/HTTP_127.6.31.148@KRBTEST.COM.keytab.
I20260927 10:50:41.198635 6270 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh8OKcT/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/postgres -l logfile start
2026-09-27 10:50:43.879 UTC [17068] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:50:43.879 UTC [17068] LOG: listening on IPv4 address "127.6.31.148", port 47055
2026-09-27 10:50:43.899 UTC [17068] LOG: listening on Unix socket "/tmp/.s.PGSQL.47055"
2026-09-27 10:50:43.934 UTC [17071] LOG: database system was shut down at 2026-09-27 10:50:42 UTC
2026-09-27 10:50:43.939 UTC [17068] LOG: database system is ready to accept connections
I20260927 10:50:44.871775 6270 mini_postgres.cc:96] Postgres bound to 47055
2026-09-27 10:50:44.878 UTC [17079] FATAL: database "slave" does not exist
127.6.31.148:47055 - accepting connections
I20260927 10:50:44.878870 6270 mini_ranger.cc:161] Starting Ranger...
I20260927 10:50:44.893812 6270 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:50:45.433969 6270 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:50:45.434068 6270 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-admin
2026-09-27 10:50:45,723 [I] DB FLAVOR :POSTGRES
2026-09-27 10:50:45,723 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:50:45,724 [I] Checking connection..
2026-09-27 10:50:45,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:50:45,986 [I] Checking connection passed.
2026-09-27 10:50:45,986 [I] --------- Verifying version history table ---------
2026-09-27 10:50:45,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:50:46,254 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:50:46,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:50:46,485 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:50:46,485 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:50:46,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:50:46.713 UTC [17143] WARNING: there is no transaction in progress
2026-09-27 10:50:46,720 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:50:46,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:50:46,981 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:50:46,981 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:50:46,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:50:47,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:50:47,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c60x.c.gcp-upstream.internal','N') ;"
2026-09-27 10:50:47,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:50:47,985 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:50:47,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:50:48,234 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:50:48,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:50:48,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:50:48,737 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:50:48,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:50:49.755 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.772 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.790 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.806 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.825 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.845 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.904 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.911 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.928 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.940 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.952 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.968 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.979 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.986 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:49.993 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:50.001 UTC [17251] WARNING: there is no transaction in progress
2026-09-27 10:50:50,708 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:50:50,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:50:50,941 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:50:50,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:50:51,191 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:50:51,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:50:51,451 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:50:51,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:50:51,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:50:51,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c60x.c.gcp-upstream.internal';"
2026-09-27 10:50:52,186 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:50:52,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:50:52,438 [I] DB_PATCHES have already been applied
I20260927 10:50:52.444417 6270 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh8OKcT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:50:52.444485 6270 mini_ranger.cc:193] Using host: 127.6.31.148
I20260927 10:50:52.447837 6270 mini_ranger.cc:227] Ranger admin URL: http://127.6.31.148:39221
Sep 27, 2026 10:50:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:50:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:50:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:50:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:50:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:50:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:50:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:50:53.561424 6270 mini_ranger.cc:160] Time spent starting Ranger: real 8.683s user 0.002s sys 0.008s
Sep 27, 2026 10:51:15 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:51:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:51:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:51:15 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:51:16 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:51:16.962141 6270 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.31.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.31.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/rangerkms_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/rangerkms_127.6.31.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.31.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.31.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.31.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/HTTP_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.31.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/HTTP_127.6.31.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:51:17 dist-test-slave-c60x krb5kdc[17038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506277, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:51:17.064126 6270 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:51:17.074208 6270 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:51:18.089682 6270 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:51:18.089781 6270 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-kms
I20260927 10:51:18.089882 6270 mini_ranger_kms.cc:227] Using postgres at 127.6.31.148:47055
2026-09-27 10:51:18,281 [I] DB FLAVOR :POSTGRES
2026-09-27 10:51:18,281 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:51:18,282 [I] Checking connection
2026-09-27 10:51:18,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:51:18,525 [I] connection success
2026-09-27 10:51:18,525 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:51:18,525 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:51:18,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:51:18,781 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:51:18,781 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:51:18,782 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:51:18,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:47055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:51:19,045 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:51:19.575330 6270 mini_ranger_kms.cc:310] Created kms service
I20260927 10:51:19.775054 6270 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:51:19.837529 6270 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:51:19 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506277, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:51:20.255194 6270 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:51:20.255506 6270 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:51:20.255544 6270 mini_ranger_kms.cc:240] Using host: 127.6.31.148
I20260927 10:51:20.259310 6270 mini_ranger_kms.cc:276] Ranger KMS PID: 17432
I20260927 10:51:20.259459 6270 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.31.148:55849
Sep 27, 2026 10:51:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:51:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:51:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:51:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:51:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:51:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 10:51:22 dist-test-slave-c60x krb5kdc[17038](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506282, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.31.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:51:22.526186 6270 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.462s user 0.006s sys 0.169s
I20260927 10:51:22.526324 6270 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:51:22.526443 6270 mini_ranger_kms.cc:401] 127.6.31.148:55849/kms/v1/keys
Sep 27 10:51:22 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506282, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.31.148@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
Sep 27 10:51:23 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506282, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.31.148@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
Sep 27 10:51:25 dist-test-slave-c60x krb5kdc[17038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:51:25.434283 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:40973
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:40973
--ranger_config_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:55849/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:51:25.547338 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:51:25.547662 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:51:25.547739 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:51:25.551862 17482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:51:25.551949 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:51:25.551988 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:51:25.552042 17482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:51:25.552083 17482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:51:25.557874 17482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.31.148:55849/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:40973
--ranger_config_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:51:25.559051 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:51:25.560124 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:51:25.569223 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:51:25.569226 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:51:25.569286 17482 server_base.cc:1034] running on GCE node
W20260927 10:51:25.569226 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:51:25.570101 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:51:25.571280 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:51:25.572465 17482 hybrid_clock.cc:648] HybridClock initialized: now 1790506285572449 us; error 30 us; skew 500 ppm
Sep 27 10:51:25 dist-test-slave-c60x krb5kdc[17038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506285, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:51:25.585887 17482 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:51:25.586891 17482 webserver.cc:460] Webserver started at http://127.6.31.190:32781/ using document root <none> and password file <none>
I20260927 10:51:25.587522 17482 fs_manager.cc:359] Metadata directory not provided
I20260927 10:51:25.587605 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:51:25.587831 17482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:51:25 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506285, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:51:25.642683 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "d8887facddd24c4a8b6fd7049468395a"
format_stamp: "Formatted at 2026-09-27 10:51:25 on dist-test-slave-c60x"
server_key: "e03a4ec4740be7df6302378d92e2644d"
server_key_iv: "501f1631f429ddbdd527de9e51fe82a0"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:25.643446 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "d8887facddd24c4a8b6fd7049468395a"
format_stamp: "Formatted at 2026-09-27 10:51:25 on dist-test-slave-c60x"
server_key: "e03a4ec4740be7df6302378d92e2644d"
server_key_iv: "501f1631f429ddbdd527de9e51fe82a0"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:25.648310 17482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 10:51:25.665694 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:25.667416 17482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:51:25.667605 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "d8887facddd24c4a8b6fd7049468395a"
format_stamp: "Formatted at 2026-09-27 10:51:25 on dist-test-slave-c60x"
server_key: "e03a4ec4740be7df6302378d92e2644d"
server_key_iv: "501f1631f429ddbdd527de9e51fe82a0"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:25.667753 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:51:25.691583 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:51:25.694962 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:51:25.695180 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:51:25.704141 17482 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:40973
I20260927 10:51:25.704229 17549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:40973 every 8 connection(s)
I20260927 10:51:25.705305 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:51:25.708213 17550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:51:25.712900 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 17482
I20260927 10:51:25.713016 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:51:25.713281 6270 external_mini_cluster.cc:1428] Setting key ca1064ee5e21cdf549281da7b8c84e67
I20260927 10:51:25.713871 17550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a: Bootstrap starting.
I20260927 10:51:25.717469 17550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a: Neither blocks nor log segments found. Creating new log.
I20260927 10:51:25.718406 17550 log.cc:824] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:51:25.720525 17550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a: No bootstrap required, opened a new log
Sep 27 10:51:25 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:51:25.725044 17550 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8887facddd24c4a8b6fd7049468395a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40973 } }
I20260927 10:51:25.725323 17550 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:51:25.725405 17550 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8887facddd24c4a8b6fd7049468395a, State: Initialized, Role: FOLLOWER
I20260927 10:51:25.726347 17550 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8887facddd24c4a8b6fd7049468395a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40973 } }
I20260927 10:51:25.726526 17550 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:51:25.726621 17550 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:51:25.726733 17550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:51:25.727919 17550 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8887facddd24c4a8b6fd7049468395a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40973 } }
I20260927 10:51:25.728331 17550 leader_election.cc:302] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8887facddd24c4a8b6fd7049468395a; no voters:
I20260927 10:51:25.728717 17550 leader_election.cc:288] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:51:25.729092 17555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:51:25.729323 17555 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [term 1 LEADER]: Becoming Leader. State: Replica: d8887facddd24c4a8b6fd7049468395a, State: Running, Role: LEADER
I20260927 10:51:25.729686 17555 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8887facddd24c4a8b6fd7049468395a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40973 } }
I20260927 10:51:25.729982 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:25.714705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.6.31.190:40973)
0927 10:51:25.715125 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:25.715144 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:25.715185 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:51:25.716033 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:25.716055 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:25.716480 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:25.716825 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:51:25.717714 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:25.719224 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:25.720424 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:25.721093 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:25.721261 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:25.724391 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:51:25.724398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:51:25.724418 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:51:25.724451 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:51:25.726458 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:25.727102 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:25.727109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:25.727116 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:25.727252 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:25.727567 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:25.727570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:25.727572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:25.728116 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:51:25.728306 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:25.728673 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 10:51:25.728974 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260927 10:51:25.730168 17550 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:51:25.730909 17556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8887facddd24c4a8b6fd7049468395a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8887facddd24c4a8b6fd7049468395a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40973 } } }
I20260927 10:51:25.730950 17557 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8887facddd24c4a8b6fd7049468395a. Latest consensus state: current_term: 1 leader_uuid: "d8887facddd24c4a8b6fd7049468395a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8887facddd24c4a8b6fd7049468395a" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40973 } } }
I20260927 10:51:25.731041 17557 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [sys.catalog]: This master's current role is: LEADER
I20260927 10:51:25.731041 17556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a [sys.catalog]: This master's current role is: LEADER
I20260927 10:51:25.733006 17550 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:51:25.736678 17559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:51:25.741184 17559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:51:25.752267 17559 catalog_manager.cc:1347] Generated new cluster ID: 5206f622f9fd483083badecf8c32e384
I20260927 10:51:25.752344 17559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:51:25.777561 17559 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:51:25.779193 17559 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:51:25.787247 17559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a: Generated new TSK 0
I20260927 10:51:25.788311 17559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:51:26.478811 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:55849/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:40973
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:51:26.603237 17595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:51:26.603477 17595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:51:26.603519 17595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:51:26.607788 17595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:51:26.607846 17595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:51:26.607926 17595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:51:26.613456 17595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.31.148:55849/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:51:26.614521 17595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:51:26.615815 17595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:51:26.625613 17595 server_base.cc:1034] running on GCE node
W20260927 10:51:26.625598 17601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:51:26.625748 17603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:51:26.625520 17600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:51:26.626309 17595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:51:26.627095 17595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:51:26.628305 17595 hybrid_clock.cc:648] HybridClock initialized: now 1790506286628269 us; error 47 us; skew 500 ppm
Sep 27 10:51:26 dist-test-slave-c60x krb5kdc[17038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:51:26.632257 17595 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:51:26.633364 17595 webserver.cc:460] Webserver started at http://127.6.31.129:41939/ using document root <none> and password file <none>
I20260927 10:51:26.634043 17595 fs_manager.cc:359] Metadata directory not provided
I20260927 10:51:26.634126 17595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:51:26.634358 17595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:51:26 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:51:26.649711 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "40de5e4be43541039868f367ba9ee588"
format_stamp: "Formatted at 2026-09-27 10:51:26 on dist-test-slave-c60x"
server_key: "5b5b7b062eeb1b20f9301f87d911c2ee"
server_key_iv: "3a94a489a45de321d4b1ecca97b521d3"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:26.650454 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "40de5e4be43541039868f367ba9ee588"
format_stamp: "Formatted at 2026-09-27 10:51:26 on dist-test-slave-c60x"
server_key: "5b5b7b062eeb1b20f9301f87d911c2ee"
server_key_iv: "3a94a489a45de321d4b1ecca97b521d3"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:26.656302 17595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 10:51:26.684504 17612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:26.686057 17595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:51:26.686184 17595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "40de5e4be43541039868f367ba9ee588"
format_stamp: "Formatted at 2026-09-27 10:51:26 on dist-test-slave-c60x"
server_key: "5b5b7b062eeb1b20f9301f87d911c2ee"
server_key_iv: "3a94a489a45de321d4b1ecca97b521d3"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:26.686319 17595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:51:26.699460 17595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:51:26.702487 17595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:51:26.702687 17595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:51:26.703366 17595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:51:26.704301 17595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:51:26.704372 17595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:26.704423 17595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:51:26.704450 17595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:26.717319 17595 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:43073
I20260927 10:51:26.718619 17595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:51:26.720214 17725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:43073 every 8 connection(s)
I20260927 10:51:26.727720 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 17595
I20260927 10:51:26.727830 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:51:26.728132 6270 external_mini_cluster.cc:1428] Setting key 7171512c04c1310ad31a35adf33be8c4
Sep 27 10:51:26 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:51:26.733209 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:26.720608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:58919 (local address 127.6.31.190:40973)
0927 10:51:26.720921 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:26.720929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:26.721593 (+ 664us) server_negotiation.cc:407] Connection header received
0927 10:51:26.722541 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:26.722544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:26.722600 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:26.722716 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:51:26.723935 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:26.724494 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:26.725694 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:26.725923 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:26.726031 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:26.728717 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:51:26.728722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:51:26.728725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:51:26.728749 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:51:26.730585 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:26.731251 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:26.731257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:26.731262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:26.731350 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:26.731768 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:26.731774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:26.731778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:26.732041 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:51:26.732131 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:26.732822 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 10:51:26.732984 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20260927 10:51:26.734856 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:26.721028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:40973 (local address 127.6.31.129:58919)
0927 10:51:26.721451 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:26.721502 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:26.722325 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:26.722884 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:26.722890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:26.723236 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:51:26.723795 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:26.723808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:26.724618 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:26.724622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:26.725568 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:26.725578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:26.726066 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:26.726071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:26.726237 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:26.726895 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:51:26.726911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:51:26.728526 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:51:26.730709 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:51:26.730718 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:51:26.730736 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:51:26.731134 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:51:26.731466 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:51:26.731471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:51:26.731476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:51:26.731654 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:51:26.732200 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:51:26.732209 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:51:26.732540 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 10:51:26.733430 (+ 890us) client_negotiation.cc:240] Negotiation successful
0927 10:51:26.733768 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":126,"threads_started":1}
I20260927 10:51:26.736234 17726 heartbeater.cc:347] Connected to a master server at 127.6.31.190:40973
I20260927 10:51:26.736522 17726 heartbeater.cc:464] Registering TS with master...
I20260927 10:51:26.737149 17726 heartbeater.cc:511] Master 127.6.31.190:40973 requested a full tablet report, sending...
I20260927 10:51:26.738688 17514 ts_manager.cc:194] Registered new tserver with Master: 40de5e4be43541039868f367ba9ee588 (127.6.31.129:43073)
I20260927 10:51:26.740082 17514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:58919
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:51:26.770560 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:55849/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:40973
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:51:26 dist-test-slave-c60x krb5kdc[17038](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 10:51:26.944389 17734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:51:26.944672 17734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:51:26.944752 17734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:51:26.949064 17734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:51:26.949149 17734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:51:26.949255 17734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:51:26.954998 17734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.31.148:55849/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:51:26.956136 17734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:51:26.957197 17734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:51:26.972029 17740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:51:26.972029 17743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:51:26.972532 17741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:51:26.973088 17734 server_base.cc:1034] running on GCE node
I20260927 10:51:26.973572 17734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:51:26.974529 17734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:51:26.977795 17734 hybrid_clock.cc:648] HybridClock initialized: now 1790506286977778 us; error 46 us; skew 500 ppm
Sep 27 10:51:26 dist-test-slave-c60x krb5kdc[17038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:51:26.981810 17734 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:51:26.982903 17734 webserver.cc:460] Webserver started at http://127.6.31.130:44041/ using document root <none> and password file <none>
I20260927 10:51:26.983680 17734 fs_manager.cc:359] Metadata directory not provided
I20260927 10:51:26.983771 17734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:51:26.984055 17734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:51:26 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:51:27.001456 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "a5c1741de1b04da19f929a96ca845251"
format_stamp: "Formatted at 2026-09-27 10:51:26 on dist-test-slave-c60x"
server_key: "4c70fa59ae99e25f008e8fbe2a8ebd6e"
server_key_iv: "e3a989fcad12059b7dca32abf73d92e4"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:27.002560 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "a5c1741de1b04da19f929a96ca845251"
format_stamp: "Formatted at 2026-09-27 10:51:26 on dist-test-slave-c60x"
server_key: "4c70fa59ae99e25f008e8fbe2a8ebd6e"
server_key_iv: "e3a989fcad12059b7dca32abf73d92e4"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:27.008731 17734 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 10:51:27.034543 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:27.036629 17734 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.002s
I20260927 10:51:27.036791 17734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "a5c1741de1b04da19f929a96ca845251"
format_stamp: "Formatted at 2026-09-27 10:51:26 on dist-test-slave-c60x"
server_key: "4c70fa59ae99e25f008e8fbe2a8ebd6e"
server_key_iv: "e3a989fcad12059b7dca32abf73d92e4"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:27.036964 17734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:51:27.056519 17734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:51:27.060087 17734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:51:27.060287 17734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:51:27.061007 17734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:51:27.062031 17734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:51:27.062098 17734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:27.062160 17734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:51:27.062191 17734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:27.072976 17734 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:44597
I20260927 10:51:27.074173 17734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:51:27.081817 17863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:44597 every 8 connection(s)
I20260927 10:51:27.082901 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 17734
I20260927 10:51:27.083020 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:51:27.083451 6270 external_mini_cluster.cc:1428] Setting key 665ad07384b3c8752aa4a59400a49744
Sep 27 10:51:27 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:51:27.109745 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.084169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:39473 (local address 127.6.31.190:40973)
0927 10:51:27.084450 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.084455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.087523 (+ 3068us) server_negotiation.cc:407] Connection header received
0927 10:51:27.088684 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.088687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.088735 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.088784 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:51:27.090618 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.091517 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.096085 (+ 4568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.096471 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.096558 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.105177 (+ 8619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:51:27.105182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:51:27.105184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:51:27.105216 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:51:27.107138 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:27.107830 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:27.107833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:27.107835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:27.107935 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:27.108399 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:27.108402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:27.108404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:27.108638 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:51:27.108719 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.109429 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.109569 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260927 10:51:27.111683 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.084325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:40973 (local address 127.6.31.130:39473)
0927 10:51:27.087379 (+ 3054us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.087435 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.088452 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.089166 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:27.089175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:27.089669 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:51:27.090381 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.090401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.091674 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.091678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.093068 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.093079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.096785 (+ 3706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.096789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.097016 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.097748 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:51:27.097771 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:51:27.104906 (+ 7135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:51:27.107305 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:51:27.107312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:51:27.107332 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:51:27.107711 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:51:27.108055 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:51:27.108058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:51:27.108060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:51:27.108258 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:51:27.108788 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:51:27.108794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:51:27.109151 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 10:51:27.110247 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0927 10:51:27.110607 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2821,"thread_start_us":171,"threads_started":1}
I20260927 10:51:27.113220 17864 heartbeater.cc:347] Connected to a master server at 127.6.31.190:40973
I20260927 10:51:27.113449 17864 heartbeater.cc:464] Registering TS with master...
I20260927 10:51:27.113996 17864 heartbeater.cc:511] Master 127.6.31.190:40973 requested a full tablet report, sending...
I20260927 10:51:27.115118 17514 ts_manager.cc:194] Registered new tserver with Master: a5c1741de1b04da19f929a96ca845251 (127.6.31.130:44597)
I20260927 10:51:27.115856 17514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:39473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:51:27.150687 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:55849/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:40973
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:51:27.304723 17869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:51:27.305126 17869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:51:27.305200 17869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:51:27.311906 17869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:51:27.311993 17869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:51:27.312083 17869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:51:27.320391 17869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.31.148:55849/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:51:27.321795 17869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:51:27.323177 17869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:51:27.336403 17874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:51:27.336759 17877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:51:27.337983 17869 server_base.cc:1034] running on GCE node
Sep 27 10:51:27 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
W20260927 10:51:27.338948 17875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:51:27.339548 17869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:51:27.340322 17869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:51:27.351292 17869 hybrid_clock.cc:648] HybridClock initialized: now 1790506287351234 us; error 80 us; skew 500 ppm
Sep 27 10:51:27 dist-test-slave-c60x krb5kdc[17038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506287, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:51:27.355301 17869 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:51:27.356563 17869 webserver.cc:460] Webserver started at http://127.6.31.131:44575/ using document root <none> and password file <none>
I20260927 10:51:27.357264 17869 fs_manager.cc:359] Metadata directory not provided
I20260927 10:51:27.357342 17869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:51:27.357570 17869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:51:27 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506287, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:51:27.380657 17869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "67298917220d4cdfb9b2476101031ae0"
format_stamp: "Formatted at 2026-09-27 10:51:27 on dist-test-slave-c60x"
server_key: "75d346147509c0a43fe49d124f2bff2e"
server_key_iv: "fe160e481a794f21feabbd62e95b9394"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:27.381428 17869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "67298917220d4cdfb9b2476101031ae0"
format_stamp: "Formatted at 2026-09-27 10:51:27 on dist-test-slave-c60x"
server_key: "75d346147509c0a43fe49d124f2bff2e"
server_key_iv: "fe160e481a794f21feabbd62e95b9394"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:27.393651 17869 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.000s
I20260927 10:51:27.413721 17884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:27.415624 17869 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 10:51:27.415839 17869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "67298917220d4cdfb9b2476101031ae0"
format_stamp: "Formatted at 2026-09-27 10:51:27 on dist-test-slave-c60x"
server_key: "75d346147509c0a43fe49d124f2bff2e"
server_key_iv: "fe160e481a794f21feabbd62e95b9394"
server_key_version: "kuduclusterkey@0"
I20260927 10:51:27.416029 17869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:51:27.447562 17869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:51:27.451310 17869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:51:27.451571 17869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:51:27.452297 17869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:51:27.453317 17869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:51:27.453438 17869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:27.453543 17869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:51:27.453605 17869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:51:27.474418 17869 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:40837
I20260927 10:51:27.475682 17869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:51:27.476934 17997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:40837 every 8 connection(s)
I20260927 10:51:27.480532 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 17869
I20260927 10:51:27.480667 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:51:27.480990 6270 external_mini_cluster.cc:1428] Setting key 5ff96c3e5f23ea8e15ceb7386501d504
Sep 27 10:51:27 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506287, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:51:27.494273 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.480200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:56639 (local address 127.6.31.190:40973)
0927 10:51:27.480317 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.480321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.481369 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 10:51:27.482569 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.482572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.482622 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.482670 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:51:27.484232 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.485063 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.486703 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.487044 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.487136 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.490117 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:51:27.490121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:51:27.490123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:51:27.490150 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:51:27.491932 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:27.492566 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:27.492570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:27.492572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:27.492657 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:27.493036 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:27.493039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:27.493041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:27.493248 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:51:27.493314 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.493982 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.494107 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:51:27.496165 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.479056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:40973 (local address 127.6.31.131:56639)
0927 10:51:27.481210 (+ 2154us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.481266 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.482332 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.483070 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:27.483098 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:27.483447 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:51:27.484075 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.484091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.485215 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.485218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.486561 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.486576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.487327 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.487331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.487536 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.488218 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:51:27.488238 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:51:27.489933 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:51:27.492075 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:51:27.492081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:51:27.492098 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:51:27.492450 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:51:27.492760 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:51:27.492763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:51:27.492765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:51:27.492932 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:51:27.493390 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:51:27.493396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:51:27.493713 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 10:51:27.494731 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0927 10:51:27.495075 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1864,"thread_start_us":135,"threads_started":1}
I20260927 10:51:27.497618 17998 heartbeater.cc:347] Connected to a master server at 127.6.31.190:40973
I20260927 10:51:27.497845 17998 heartbeater.cc:464] Registering TS with master...
I20260927 10:51:27.498371 17998 heartbeater.cc:511] Master 127.6.31.190:40973 requested a full tablet report, sending...
I20260927 10:51:27.499485 17514 ts_manager.cc:194] Registered new tserver with Master: 67298917220d4cdfb9b2476101031ae0 (127.6.31.131:40837)
I20260927 10:51:27.500236 17514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:56639
I20260927 10:51:27.508080 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:51:27.522186 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.510725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.6.31.190:40973)
0927 10:51:27.510855 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.510860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.511507 (+ 647us) server_negotiation.cc:407] Connection header received
0927 10:51:27.511729 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.511733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.511785 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.511841 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:51:27.513014 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.513914 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.515332 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.515728 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.515821 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.517967 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:51:27.517972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:51:27.517975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:51:27.518005 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:51:27.520055 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:27.520635 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:27.520639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:27.520642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:27.520735 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:51:27.521081 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:51:27.521085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:51:27.521087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:51:27.521330 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:51:27.521463 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.521859 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.521998 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:51:27.526242 17514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:51:27.530884 17514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:51:27.559266 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.548863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:40837 (local address 127.0.0.1:40600)
0927 10:51:27.551128 (+ 2265us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.551180 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.551298 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.552510 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:27.552516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:27.552538 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:51:27.552692 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.552699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.556457 (+ 3758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.556461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.557638 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.557647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.558745 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.558748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.558854 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.558880 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:51:27.558935 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 10:51:27.559108 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2076,"thread_start_us":99,"threads_started":1}
I20260927 10:51:27.559584 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.549087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.6.31.130:44597)
0927 10:51:27.549384 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.549401 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.551632 (+ 2231us) server_negotiation.cc:407] Connection header received
0927 10:51:27.551735 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.551742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.551995 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.553320 (+ 1325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:51:27.553483 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.554761 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.557558 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.558497 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.559073 (+ 576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.559257 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.559313 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.559414 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":118,"threads_started":1}
I20260927 10:51:27.560092 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.548851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:44597 (local address 127.0.0.1:39440)
0927 10:51:27.551508 (+ 2657us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.551519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.551597 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.553016 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:27.553019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:27.553040 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:51:27.553150 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.553158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.554942 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.554947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.556551 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.556563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.558653 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.558658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.558786 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.558833 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 10:51:27.558936 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 10:51:27.559968 (+ 1032us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2589,"mutex_wait_us":1907,"thread_start_us":79,"threads_started":1}
I20260927 10:51:27.561671 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.549693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.6.31.131:40837)
0927 10:51:27.552065 (+ 2372us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.552076 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.552097 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:51:27.552162 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.552167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.552332 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.552840 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:51:27.552961 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.554135 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.557810 (+ 3675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.558609 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.561226 (+ 2617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.561380 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.561442 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.561532 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2275,"thread_start_us":149,"threads_started":1}
I20260927 10:51:27.563642 17798 tablet_service.cc:1467] Processing CreateTablet for tablet d9dc511f4a0d4f97840efd8f5de5a5e7 (DEFAULT_TABLE table=test-table [id=df2e1b10e8124eecb547d4900eef79b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:51:27.564972 17798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9dc511f4a0d4f97840efd8f5de5a5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:51:27.567104 17929 tablet_service.cc:1467] Processing CreateTablet for tablet d9dc511f4a0d4f97840efd8f5de5a5e7 (DEFAULT_TABLE table=test-table [id=df2e1b10e8124eecb547d4900eef79b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:51:27.568216 17929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9dc511f4a0d4f97840efd8f5de5a5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:51:27.576686 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.550676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:43073 (local address 127.0.0.1:36652)
0927 10:51:27.551760 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.551770 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.551829 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.554805 (+ 2976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:27.554808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:27.554824 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:51:27.554905 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.554911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.561925 (+ 7014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.561928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.563207 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.563218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.576166 (+ 12948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.576170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.576294 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.576326 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 10:51:27.576417 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:51:27.576507 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":86,"threads_started":1}
I20260927 10:51:27.577626 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.550889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.6.31.129:43073)
0927 10:51:27.554388 (+ 3499us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.554394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.554427 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:51:27.554494 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.554498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.554665 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.555044 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:51:27.555211 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.556283 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.563377 (+ 7094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.564201 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.577031 (+ 12830us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.577238 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.577298 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.577403 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3395,"thread_start_us":102,"threads_started":1}
I20260927 10:51:27.579891 18017 tablet_bootstrap.cc:492] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: Bootstrap starting.
I20260927 10:51:27.581025 17652 tablet_service.cc:1467] Processing CreateTablet for tablet d9dc511f4a0d4f97840efd8f5de5a5e7 (DEFAULT_TABLE table=test-table [id=df2e1b10e8124eecb547d4900eef79b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:51:27.582284 17652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9dc511f4a0d4f97840efd8f5de5a5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:51:27.582647 18017 tablet_bootstrap.cc:654] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: Neither blocks nor log segments found. Creating new log.
I20260927 10:51:27.583488 18017 log.cc:824] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:51:27.585510 18017 tablet_bootstrap.cc:492] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: No bootstrap required, opened a new log
I20260927 10:51:27.585707 18017 ts_tablet_manager.cc:1397] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:51:27.586339 18016 tablet_bootstrap.cc:492] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: Bootstrap starting.
I20260927 10:51:27.589382 18016 tablet_bootstrap.cc:654] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: Neither blocks nor log segments found. Creating new log.
I20260927 10:51:27.589637 18017 raft_consensus.cc:348] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.589906 18017 raft_consensus.cc:374] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:51:27.589973 18017 raft_consensus.cc:729] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67298917220d4cdfb9b2476101031ae0, State: Initialized, Role: FOLLOWER
I20260927 10:51:27.590504 18017 consensus_queue.cc:260] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.591596 18017 ts_tablet_manager.cc:1428] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:51:27.591933 17998 heartbeater.cc:503] Master 127.6.31.190:40973 was elected leader, sending a full tablet report...
I20260927 10:51:27.595588 18016 log.cc:824] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: Log is configured to *not* fsync() on all Append() calls
I20260927 10:51:27.596467 18020 tablet_bootstrap.cc:492] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588: Bootstrap starting.
I20260927 10:51:27.598483 18016 tablet_bootstrap.cc:492] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: No bootstrap required, opened a new log
I20260927 10:51:27.598780 18016 ts_tablet_manager.cc:1397] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260927 10:51:27.601210 18020 tablet_bootstrap.cc:654] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588: Neither blocks nor log segments found. Creating new log.
I20260927 10:51:27.602108 18020 log.cc:824] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588: Log is configured to *not* fsync() on all Append() calls
I20260927 10:51:27.603554 18016 raft_consensus.cc:348] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.603881 18016 raft_consensus.cc:374] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:51:27.603974 18016 raft_consensus.cc:729] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c1741de1b04da19f929a96ca845251, State: Initialized, Role: FOLLOWER
I20260927 10:51:27.604513 18016 consensus_queue.cc:260] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.604714 18020 tablet_bootstrap.cc:492] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588: No bootstrap required, opened a new log
I20260927 10:51:27.604918 18020 ts_tablet_manager.cc:1397] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 10:51:27.605787 18016 ts_tablet_manager.cc:1428] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:51:27.606194 17864 heartbeater.cc:503] Master 127.6.31.190:40973 was elected leader, sending a full tablet report...
I20260927 10:51:27.609401 18020 raft_consensus.cc:348] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.609723 18020 raft_consensus.cc:374] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:51:27.609766 18020 raft_consensus.cc:729] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40de5e4be43541039868f367ba9ee588, State: Initialized, Role: FOLLOWER
I20260927 10:51:27.610200 18020 consensus_queue.cc:260] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.610927 17726 heartbeater.cc:503] Master 127.6.31.190:40973 was elected leader, sending a full tablet report...
I20260927 10:51:27.611284 18020 ts_tablet_manager.cc:1428] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
Sep 27 10:51:27 dist-test-slave-c60x krb5kdc[17038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
W20260927 10:51:27.720710 17727 tablet.cc:2369] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:51:27.731155 17999 tablet.cc:2369] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:51:27.751665 18024 raft_consensus.cc:482] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:51:27.751883 18024 raft_consensus.cc:504] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.753193 18024 leader_election.cc:288] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5c1741de1b04da19f929a96ca845251 (127.6.31.130:44597), 67298917220d4cdfb9b2476101031ae0 (127.6.31.131:40837)
I20260927 10:51:27.763182 18019 raft_consensus.cc:482] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:51:27.763386 18019 raft_consensus.cc:504] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.764689 18019 leader_election.cc:288] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5c1741de1b04da19f929a96ca845251 (127.6.31.130:44597), 40de5e4be43541039868f367ba9ee588 (127.6.31.129:43073)
I20260927 10:51:27.767777 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.754440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:43851 (local address 127.6.31.130:44597)
0927 10:51:27.754552 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.754556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.763654 (+ 9098us) server_negotiation.cc:407] Connection header received
0927 10:51:27.763844 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.763848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.763918 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.763967 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:51:27.764369 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.765103 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.766498 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.767115 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.767227 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.767271 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.767582 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.767655 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:51:27.767777 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.753702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:34247 (local address 127.6.31.131:40837)
0927 10:51:27.753827 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.753832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.760351 (+ 6519us) server_negotiation.cc:407] Connection header received
0927 10:51:27.760570 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.760574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.760626 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.760673 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:51:27.761088 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.761832 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.766544 (+ 4712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.767138 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.767221 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.767271 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.767582 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.767653 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:51:27.768800 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.753585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:40837 (local address 127.6.31.129:34247)
0927 10:51:27.760254 (+ 6669us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.760270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.760437 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.760829 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:27.760833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:27.760850 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:51:27.760961 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.760967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.762008 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.762011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.763296 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.763307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.767347 (+ 4040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.767350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.767435 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.767459 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:51:27.767958 (+ 499us) client_negotiation.cc:240] Negotiation successful
0927 10:51:27.768034 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6588,"thread_start_us":114,"threads_started":1}
I20260927 10:51:27.768810 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.754080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:44597 (local address 127.6.31.129:43851)
0927 10:51:27.763563 (+ 9483us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.763579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.763721 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.764109 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:27.764114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:27.764132 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:51:27.764233 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.764240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.765242 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.765246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.766369 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.766379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.767333 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.767337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.767460 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.767475 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 10:51:27.768581 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0927 10:51:27.768655 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9407,"mutex_wait_us":26,"thread_start_us":96,"threads_started":1}
I20260927 10:51:27.769397 17951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dc511f4a0d4f97840efd8f5de5a5e7" candidate_uuid: "40de5e4be43541039868f367ba9ee588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67298917220d4cdfb9b2476101031ae0" is_pre_election: true
I20260927 10:51:27.769635 17951 raft_consensus.cc:2454] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40de5e4be43541039868f367ba9ee588 in term 0.
I20260927 10:51:27.770092 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dc511f4a0d4f97840efd8f5de5a5e7" candidate_uuid: "40de5e4be43541039868f367ba9ee588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c1741de1b04da19f929a96ca845251" is_pre_election: true
I20260927 10:51:27.770335 17818 raft_consensus.cc:2454] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40de5e4be43541039868f367ba9ee588 in term 0.
I20260927 10:51:27.770797 17616 leader_election.cc:302] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40de5e4be43541039868f367ba9ee588, a5c1741de1b04da19f929a96ca845251; no voters:
I20260927 10:51:27.771128 18024 raft_consensus.cc:2781] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:51:27.771214 18024 raft_consensus.cc:482] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:51:27.771267 18024 raft_consensus.cc:3037] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:51:27.771867 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.765040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:44597 (local address 127.6.31.131:60837)
0927 10:51:27.765668 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.765682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.765760 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.768127 (+ 2367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:27.768130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:27.768153 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:51:27.768263 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.768281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.769295 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.769299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.770526 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.770537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.771487 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.771491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.771594 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.771615 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:51:27.771684 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 10:51:27.771751 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561}
I20260927 10:51:27.772626 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.765983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:60837 (local address 127.6.31.130:44597)
0927 10:51:27.767929 (+ 1946us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.767934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.767948 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:51:27.767986 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.767988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.768028 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.768104 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:51:27.768425 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.769159 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.770636 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.771259 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.771372 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.771429 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.772436 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.772505 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1884,"thread_start_us":71,"threads_started":1}
I20260927 10:51:27.772759 18024 raft_consensus.cc:504] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.772977 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dc511f4a0d4f97840efd8f5de5a5e7" candidate_uuid: "67298917220d4cdfb9b2476101031ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c1741de1b04da19f929a96ca845251" is_pre_election: true
I20260927 10:51:27.773082 17818 raft_consensus.cc:2454] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67298917220d4cdfb9b2476101031ae0 in term 0.
I20260927 10:51:27.773120 18024 leader_election.cc:288] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [CANDIDATE]: Term 1 election: Requested vote from peers a5c1741de1b04da19f929a96ca845251 (127.6.31.130:44597), 67298917220d4cdfb9b2476101031ae0 (127.6.31.131:40837)
I20260927 10:51:27.773779 17951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dc511f4a0d4f97840efd8f5de5a5e7" candidate_uuid: "40de5e4be43541039868f367ba9ee588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67298917220d4cdfb9b2476101031ae0"
I20260927 10:51:27.773914 17951 raft_consensus.cc:3037] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:51:27.774027 17888 leader_election.cc:302] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67298917220d4cdfb9b2476101031ae0, a5c1741de1b04da19f929a96ca845251; no voters:
I20260927 10:51:27.774300 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dc511f4a0d4f97840efd8f5de5a5e7" candidate_uuid: "40de5e4be43541039868f367ba9ee588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c1741de1b04da19f929a96ca845251"
I20260927 10:51:27.775107 17818 raft_consensus.cc:3037] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:51:27.775178 17951 raft_consensus.cc:2454] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40de5e4be43541039868f367ba9ee588 in term 1.
I20260927 10:51:27.776163 17818 raft_consensus.cc:2454] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40de5e4be43541039868f367ba9ee588 in term 1.
I20260927 10:51:27.776467 17616 leader_election.cc:302] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40de5e4be43541039868f367ba9ee588, a5c1741de1b04da19f929a96ca845251; no voters:
I20260927 10:51:27.776623 18019 raft_consensus.cc:2741] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 10:51:27.776772 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.766197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:48685 (local address 127.6.31.129:43073)
0927 10:51:27.766303 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.766307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.772171 (+ 5864us) server_negotiation.cc:407] Connection header received
0927 10:51:27.772214 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.772216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.772259 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.772322 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:51:27.773155 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.773887 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.775301 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.775897 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.776023 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.776069 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.776600 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.776677 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:51:27.777096 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.765288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:43073 (local address 127.6.31.131:48685)
0927 10:51:27.772007 (+ 6719us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.772020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.772093 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.772928 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:51:27.772931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:51:27.772950 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:51:27.773025 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.773030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.774018 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.774022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.775158 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:51:27.775168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.776034 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.776041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:51:27.776150 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.776177 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:51:27.776348 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 10:51:27.776442 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6667,"thread_start_us":120,"threads_started":1}
I20260927 10:51:27.777488 18024 raft_consensus.cc:2781] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:51:27.777706 18024 raft_consensus.cc:686] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [term 1 LEADER]: Becoming Leader. State: Replica: 40de5e4be43541039868f367ba9ee588, State: Running, Role: LEADER
I20260927 10:51:27.777736 17668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dc511f4a0d4f97840efd8f5de5a5e7" candidate_uuid: "67298917220d4cdfb9b2476101031ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40de5e4be43541039868f367ba9ee588" is_pre_election: true
I20260927 10:51:27.778043 18024 consensus_queue.cc:237] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } }
I20260927 10:51:27.781149 17513 catalog_manager.cc:5495] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40de5e4be43541039868f367ba9ee588 (127.6.31.129). New cstate: current_term: 1 leader_uuid: "40de5e4be43541039868f367ba9ee588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40de5e4be43541039868f367ba9ee588" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43073 } health_report { overall_health: HEALTHY } } }
I20260927 10:51:27.815063 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:51:27.803830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.6.31.129:43073)
0927 10:51:27.803957 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:51:27.803962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:51:27.804011 (+ 49us) server_negotiation.cc:407] Connection header received
0927 10:51:27.804197 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:51:27.804201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:51:27.804266 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:51:27.804395 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:51:27.804687 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.805432 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.806642 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:51:27.806954 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:51:27.807052 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:51:27.813094 (+ 6042us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:51:27.813727 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:51:27.813855 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:51:27.814336 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 10:51:27.814415 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:51:27.822001 17818 raft_consensus.cc:1261] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 1 FOLLOWER]: Refusing update from remote peer 40de5e4be43541039868f367ba9ee588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:51:27.822954 17951 raft_consensus.cc:1261] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 1 FOLLOWER]: Refusing update from remote peer 40de5e4be43541039868f367ba9ee588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:51:27.823594 18031 consensus_queue.cc:1035] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c1741de1b04da19f929a96ca845251" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:51:27.824067 18031 consensus_queue.cc:1035] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 40de5e4be43541039868f367ba9ee588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67298917220d4cdfb9b2476101031ae0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:51:27.836730 17865 tablet.cc:2369] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:51:27.837174 18034 mvcc.cc:205] Tried to move back new op lower bound from 7333913754910806016 to 7333913754741788672. Current Snapshot: MvccSnapshot[applied={T|T < 7333913754910806016}]
I20260927 10:51:27.841413 18035 mvcc.cc:205] Tried to move back new op lower bound from 7333913754910806016 to 7333913754741788672. Current Snapshot: MvccSnapshot[applied={T|T < 7333913754910806016}]
I20260927 10:51:27.841599 18036 mvcc.cc:205] Tried to move back new op lower bound from 7333913754910806016 to 7333913754741788672. Current Snapshot: MvccSnapshot[applied={T|T < 7333913754910806016}]
I20260927 10:51:27.852030 17513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:51:27.852219 17513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:51:27.854454 17513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8887facddd24c4a8b6fd7049468395a: Sending DeleteTablet for 3 replicas of tablet d9dc511f4a0d4f97840efd8f5de5a5e7
I20260927 10:51:27.855304 17798 tablet_service.cc:1514] Processing DeleteTablet for tablet d9dc511f4a0d4f97840efd8f5de5a5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:51:27 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:39440
I20260927 10:51:27.855782 18043 tablet_replica.cc:330] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: stopping tablet replica
I20260927 10:51:27.855932 17929 tablet_service.cc:1514] Processing DeleteTablet for tablet d9dc511f4a0d4f97840efd8f5de5a5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:51:27 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:40600
I20260927 10:51:27.856251 18043 raft_consensus.cc:2229] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:51:27.856521 17652 tablet_service.cc:1514] Processing DeleteTablet for tablet d9dc511f4a0d4f97840efd8f5de5a5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:51:27 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:36652
I20260927 10:51:27.856686 18043 raft_consensus.cc:2258] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:51:27.857697 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 17595
I20260927 10:51:27.858347 18043 ts_tablet_manager.cc:1905] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:51:27.861739 18043 ts_tablet_manager.cc:1918] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:51:27.861912 18043 log.cc:1196] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/d9dc511f4a0d4f97840efd8f5de5a5e7
I20260927 10:51:27.862298 18043 ts_tablet_manager.cc:1939] T d9dc511f4a0d4f97840efd8f5de5a5e7 P a5c1741de1b04da19f929a96ca845251: Deleting consensus metadata
I20260927 10:51:27.863639 17501 catalog_manager.cc:4841] TS a5c1741de1b04da19f929a96ca845251 (127.6.31.130:44597): tablet d9dc511f4a0d4f97840efd8f5de5a5e7 (table test-table [id=df2e1b10e8124eecb547d4900eef79b4]) successfully deleted
I20260927 10:51:27.865871 18044 tablet_replica.cc:330] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: stopping tablet replica
I20260927 10:51:27.866165 18044 raft_consensus.cc:2229] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:51:27.866467 18044 raft_consensus.cc:2258] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 10:51:27.867122 17501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:43073 (error 108)
I20260927 10:51:27.868412 18044 ts_tablet_manager.cc:1905] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:51:27.870923 18044 ts_tablet_manager.cc:1918] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:51:27.871030 18044 log.cc:1196] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/d9dc511f4a0d4f97840efd8f5de5a5e7
I20260927 10:51:27.871384 18044 ts_tablet_manager.cc:1939] T d9dc511f4a0d4f97840efd8f5de5a5e7 P 67298917220d4cdfb9b2476101031ae0: Deleting consensus metadata
W20260927 10:51:27.872601 18010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:51:27.867791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:43073 (local address 127.0.0.1:36670)
0927 10:51:27.867925 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 10:51:27.867943 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:51:27.868040 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:51:27.872483 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:43073: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 10:51:27.872795 17501 catalog_manager.cc:4584] TS 40de5e4be43541039868f367ba9ee588 (127.6.31.129:43073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9dc511f4a0d4f97840efd8f5de5a5e7: Network error: Client connection negotiation failed: client connection to 127.6.31.129:43073: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:51:27.873934 17498 catalog_manager.cc:4841] TS 67298917220d4cdfb9b2476101031ae0 (127.6.31.131:40837): tablet d9dc511f4a0d4f97840efd8f5de5a5e7 (table test-table [id=df2e1b10e8124eecb547d4900eef79b4]) successfully deleted
I20260927 10:51:27.874275 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 17734
I20260927 10:51:27.883455 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 17869
I20260927 10:51:27.892259 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 17482
I20260927 10:51:27.898864 6270 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:51:28.243826 6270 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:51:28.243957 6270 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:51:28.589063 6270 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:51:28.589 UTC [17068] LOG: received smart shutdown request
2026-09-27 10:51:28.593 UTC [17068] LOG: background worker "logical replication launcher" (PID 17077) exited with exit code 1
2026-09-27 10:51:28.593 UTC [17072] LOG: shutting down
2026-09-27 10:51:28.774 UTC [17068] LOG: database system is shut down
2026-09-27T10:51:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48927 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:51:28 dist-test-slave-c60x krb5kdc[18055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:51:28 dist-test-slave-c60x krb5kdc[18055](info): set up 2 sockets
Sep 27 10:51:28 dist-test-slave-c60x krb5kdc[18055](info): commencing operation
krb5kdc: starting...
W20260927 10:51:29.943255 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:51:30 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:51:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:51:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.31.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.31.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/rangeradmin_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/rangeradmin_127.6.31.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.31.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.31.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/rangerlookup_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/rangerlookup_127.6.31.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.31.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.31.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/HTTP_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/HTTP_127.6.31.148@KRBTEST.COM.keytab.
I20260927 10:51:30.122192 6270 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh8OKcT/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/postgres -l logfile start
2026-09-27 10:51:32.812 UTC [18085] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:51:32.812 UTC [18085] LOG: listening on IPv4 address "127.6.31.148", port 36963
2026-09-27 10:51:32.816 UTC [18085] LOG: listening on Unix socket "/tmp/.s.PGSQL.36963"
2026-09-27 10:51:32.852 UTC [18088] LOG: database system was shut down at 2026-09-27 10:51:31 UTC
2026-09-27 10:51:32.857 UTC [18085] LOG: database system is ready to accept connections
I20260927 10:51:33.798949 6270 mini_postgres.cc:96] Postgres bound to 36963
2026-09-27 10:51:33.805 UTC [18096] FATAL: database "slave" does not exist
127.6.31.148:36963 - accepting connections
I20260927 10:51:33.805716 6270 mini_ranger.cc:161] Starting Ranger...
I20260927 10:51:33.820979 6270 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:51:34.360231 6270 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:51:34.360316 6270 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-admin
2026-09-27 10:51:34,634 [I] DB FLAVOR :POSTGRES
2026-09-27 10:51:34,634 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:51:34,634 [I] Checking connection..
2026-09-27 10:51:34,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:51:34,878 [I] Checking connection passed.
2026-09-27 10:51:34,878 [I] --------- Verifying version history table ---------
2026-09-27 10:51:34,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:51:35,119 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:51:35,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:51:35,356 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:51:35,356 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:51:35,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:51:35.583 UTC [18160] WARNING: there is no transaction in progress
2026-09-27 10:51:35,589 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:51:35,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:51:35,833 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:51:35,833 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:51:35,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:51:36,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:51:36,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c60x.c.gcp-upstream.internal','N') ;"
2026-09-27 10:51:36,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:51:36,809 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:51:36,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:51:37,054 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:51:37,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:51:37,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:51:37,549 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:51:37,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:51:38.546 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.562 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.578 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.594 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.610 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.629 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.684 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.690 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.704 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.714 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.725 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.739 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.749 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.755 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.762 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:38.769 UTC [18268] WARNING: there is no transaction in progress
2026-09-27 10:51:39,430 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:51:39,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:51:39,658 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:51:39,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:51:39,918 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:51:39,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:51:40,187 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:51:40,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:51:40,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:51:40,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c60x.c.gcp-upstream.internal';"
2026-09-27 10:51:40,908 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:51:40,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:51:41,153 [I] DB_PATCHES have already been applied
I20260927 10:51:41.159075 6270 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh8OKcT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:51:41.159184 6270 mini_ranger.cc:193] Using host: 127.6.31.148
I20260927 10:51:41.162621 6270 mini_ranger.cc:227] Ranger admin URL: http://127.6.31.148:33015
Sep 27, 2026 10:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:51:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:51:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:51:42.298202 6270 mini_ranger.cc:160] Time spent starting Ranger: real 8.492s user 0.000s sys 0.010s
Sep 27, 2026 10:52:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:52:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 10:52:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:52:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:52:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:52:03.263072 6270 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.31.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.31.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/rangerkms_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.31.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/rangerkms_127.6.31.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.31.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.31.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.31.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/HTTP_127.6.31.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.31.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/HTTP_127.6.31.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:52:03 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506323, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:52:03.354610 6270 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:52:03.364588 6270 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:52:04.338337 6270 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:52:04.338423 6270 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-kms
I20260927 10:52:04.338446 6270 mini_ranger_kms.cc:227] Using postgres at 127.6.31.148:36963
2026-09-27 10:52:04,513 [I] DB FLAVOR :POSTGRES
2026-09-27 10:52:04,514 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:52:04,514 [I] Checking connection
2026-09-27 10:52:04,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:52:04,755 [I] connection success
2026-09-27 10:52:04,755 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:52:04,755 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:52:04,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:52:04,999 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:52:04,999 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:52:04,999 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:52:04,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.31.148:36963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:52:05,285 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:52:05.724349 6270 mini_ranger_kms.cc:310] Created kms service
I20260927 10:52:05.952169 6270 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:52:05.996884 6270 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:52:05 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506323, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:06.408505 6270 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:52:06.408653 6270 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh8OKcT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh8OKcT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:52:06.408699 6270 mini_ranger_kms.cc:240] Using host: 127.6.31.148
I20260927 10:52:06.412770 6270 mini_ranger_kms.cc:276] Ranger KMS PID: 18448
I20260927 10:52:06.412854 6270 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.31.148:45363
Sep 27, 2026 10:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 10:52:08 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506328, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.31.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:08.696135 6270 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.342s user 0.009s sys 0.150s
I20260927 10:52:08.696262 6270 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:52:08.696339 6270 mini_ranger_kms.cc:401] 127.6.31.148:45363/kms/v1/keys
Sep 27 10:52:09 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506328, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.31.148@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
Sep 27 10:52:09 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506328, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.31.148@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
Sep 27 10:52:11 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:52:11.750808 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46479
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:46479
--ranger_config_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:45363/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:11.860631 18499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:11.860914 18499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:11.860954 18499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:11.865017 18499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:11.865077 18499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:11.865095 18499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:11.865111 18499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:11.865170 18499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:11.870651 18499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.31.148:45363/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:46479
--ranger_config_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:11.871762 18499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:11.872860 18499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:52:11.882114 18499 server_base.cc:1034] running on GCE node
W20260927 10:52:11.882155 18507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:11.882174 18505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:11.882174 18504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:11.882941 18499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:11.884096 18499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:11.885278 18499 hybrid_clock.cc:648] HybridClock initialized: now 1790506331885256 us; error 39 us; skew 500 ppm
Sep 27 10:52:11 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:11.888187 18499 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:52:11.889042 18499 webserver.cc:460] Webserver started at http://127.6.31.190:39179/ using document root <none> and password file <none>
I20260927 10:52:11.889585 18499 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:11.889663 18499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:11.889923 18499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:52:11 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:11.950053 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "dde4daf24ed245dbb23620eb5f3b7de9"
format_stamp: "Formatted at 2026-09-27 10:52:11 on dist-test-slave-c60x"
server_key: "c84a7047e105335aad17f7f3bf2c933d"
server_key_iv: "3146df4d4d850ae0b93ee9af7af731ae"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:11.950847 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "dde4daf24ed245dbb23620eb5f3b7de9"
format_stamp: "Formatted at 2026-09-27 10:52:11 on dist-test-slave-c60x"
server_key: "c84a7047e105335aad17f7f3bf2c933d"
server_key_iv: "3146df4d4d850ae0b93ee9af7af731ae"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:11.954767 18499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:52:11.973381 18514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:11.974889 18499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:52:11.975054 18499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "dde4daf24ed245dbb23620eb5f3b7de9"
format_stamp: "Formatted at 2026-09-27 10:52:11 on dist-test-slave-c60x"
server_key: "c84a7047e105335aad17f7f3bf2c933d"
server_key_iv: "3146df4d4d850ae0b93ee9af7af731ae"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:11.975257 18499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:12.025501 18499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:12.028913 18499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:12.029136 18499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:12.037133 18499 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:46479
I20260927 10:52:12.037165 18566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:46479 every 8 connection(s)
I20260927 10:52:12.038108 18499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:52:12.038718 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 18499
I20260927 10:52:12.038833 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:52:12.039153 6270 external_mini_cluster.cc:1428] Setting key e2605a6dcb2f1970873dddd99506b917
I20260927 10:52:12.041339 18567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:12.047749 18567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Bootstrap starting.
Sep 27 10:52:12 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:12.050508 18567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:12.051405 18567 log.cc:824] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:12.053431 18567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: No bootstrap required, opened a new log
I20260927 10:52:12.056468 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:12.040582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.6.31.190:46479)
0927 10:52:12.041259 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:12.041278 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:12.041321 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:52:12.042241 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:12.042263 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:12.042647 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:12.043002 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:12.044048 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:12.045471 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:12.046755 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:12.047444 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:12.047601 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:12.050570 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:12.050577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:12.050595 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:12.050628 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:12.053331 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:12.053862 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:12.053868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:12.053874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:12.054001 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:12.054315 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:12.054318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:12.054320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:12.054799 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:12.054981 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:12.055304 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 10:52:12.055573 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":143,"threads_started":1}
I20260927 10:52:12.056949 18567 raft_consensus.cc:348] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } }
I20260927 10:52:12.057190 18567 raft_consensus.cc:374] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:12.057263 18567 raft_consensus.cc:729] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde4daf24ed245dbb23620eb5f3b7de9, State: Initialized, Role: FOLLOWER
I20260927 10:52:12.057741 18567 consensus_queue.cc:260] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } }
I20260927 10:52:12.057893 18567 raft_consensus.cc:388] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:52:12.057981 18567 raft_consensus.cc:482] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:52:12.058073 18567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:12.059376 18567 raft_consensus.cc:504] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } }
I20260927 10:52:12.059723 18567 leader_election.cc:302] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dde4daf24ed245dbb23620eb5f3b7de9; no voters:
I20260927 10:52:12.060071 18567 leader_election.cc:288] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:52:12.060190 18572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:52:12.060406 18572 raft_consensus.cc:686] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 1 LEADER]: Becoming Leader. State: Replica: dde4daf24ed245dbb23620eb5f3b7de9, State: Running, Role: LEADER
I20260927 10:52:12.060724 18572 consensus_queue.cc:237] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } }
I20260927 10:52:12.061436 18567 sys_catalog.cc:564] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:52:12.062039 18572 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dde4daf24ed245dbb23620eb5f3b7de9. Latest consensus state: current_term: 1 leader_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } } }
I20260927 10:52:12.062772 18572 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:12.062660 18573 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } } }
I20260927 10:52:12.063133 18573 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:12.063445 18582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:52:12.063687 18567 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:52:12.069262 18582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:52:12.081372 18582 catalog_manager.cc:1347] Generated new cluster ID: 34ad72aaf08e4579b3e15c7189058735
I20260927 10:52:12.081452 18582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:52:12.094519 18582 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:52:12.095981 18582 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:52:12.105795 18582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Generated new TSK 0
I20260927 10:52:12.106493 18582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:52:12.827751 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46479
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:12.974756 18612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:12.975126 18612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:12.975243 18612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:12.980103 18612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:12.980182 18612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:12.980288 18612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:52:12.985800 18612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.31.148:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:12.986866 18612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:12.987943 18612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:12.998023 18619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:12.998260 18620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:12.998482 18612 server_base.cc:1034] running on GCE node
W20260927 10:52:12.998211 18622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:12.999159 18612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:12.999894 18612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:13.001083 18612 hybrid_clock.cc:648] HybridClock initialized: now 1790506333001043 us; error 47 us; skew 500 ppm
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:13.005460 18612 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:52:13.006489 18612 webserver.cc:460] Webserver started at http://127.6.31.129:40635/ using document root <none> and password file <none>
I20260927 10:52:13.007061 18612 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:13.007257 18612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:13.007494 18612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:13.024377 18612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "cf545a05264d4438964dd9e0bfd27319"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "15337537425d4462bbfbe83cc899528e"
server_key_iv: "c40ca923b6dd877bca4f9d6477ba3512"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:13.025019 18612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "cf545a05264d4438964dd9e0bfd27319"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "15337537425d4462bbfbe83cc899528e"
server_key_iv: "c40ca923b6dd877bca4f9d6477ba3512"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:13.029378 18612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:52:13.070215 18630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:13.071913 18612 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:52:13.072126 18612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "cf545a05264d4438964dd9e0bfd27319"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "15337537425d4462bbfbe83cc899528e"
server_key_iv: "c40ca923b6dd877bca4f9d6477ba3512"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:13.072362 18612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:13.098380 18612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:13.102088 18612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:13.102345 18612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:13.103152 18612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:13.104235 18612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:13.104339 18612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:13.104460 18612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:13.104539 18612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:13.118974 18612 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:36283
I20260927 10:52:13.120309 18612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:52:13.121914 18744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:36283 every 8 connection(s)
I20260927 10:52:13.130218 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 18612
I20260927 10:52:13.130367 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:52:13.130653 6270 external_mini_cluster.cc:1428] Setting key 3f195f1d68776e4891d1c216e2b378a4
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:52:13.147915 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:13.129094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:36761 (local address 127.6.31.190:46479)
0927 10:52:13.130943 (+ 1849us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:13.130948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:13.132710 (+ 1762us) server_negotiation.cc:407] Connection header received
0927 10:52:13.132757 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:13.132760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:13.132814 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:13.132896 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:13.134454 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.135353 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.137085 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.137447 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.137529 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:13.142755 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:13.142759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:13.142762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:13.142792 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:13.145069 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:13.145806 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:13.145812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:13.145815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:13.145914 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:13.146376 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:13.146383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:13.146384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:13.146692 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:13.146778 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:13.147586 (+ 808us) server_negotiation.cc:299] Negotiation successful
0927 10:52:13.147732 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1755,"thread_start_us":97,"threads_started":1}
I20260927 10:52:13.149513 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:13.129641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46479 (local address 127.6.31.129:36761)
0927 10:52:13.131275 (+ 1634us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:13.131343 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:13.132389 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:13.133205 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:13.133214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:13.133606 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:13.134281 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:13.134299 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.135522 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.135526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:13.136944 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:13.136954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.137759 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.137762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:13.137957 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:13.140708 (+ 2751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:13.140730 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:13.142514 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:13.145238 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:13.145247 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:13.145266 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:13.145673 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:13.146038 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:13.146044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:13.146049 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:13.146248 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:13.146877 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:13.146885 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:13.147288 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 10:52:13.148106 (+ 818us) client_negotiation.cc:240] Negotiation successful
0927 10:52:13.148462 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1359,"thread_start_us":168,"threads_started":1}
I20260927 10:52:13.151259 18745 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46479
I20260927 10:52:13.151521 18745 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:13.152091 18745 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
I20260927 10:52:13.153883 18524 ts_manager.cc:194] Registered new tserver with Master: cf545a05264d4438964dd9e0bfd27319 (127.6.31.129:36283)
I20260927 10:52:13.155543 18524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:36761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:52:13.180290 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46479
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:13.332285 18751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:13.332629 18751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:13.332696 18751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:13.338925 18751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:13.339053 18751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:13.339187 18751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:52:13.347072 18751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.31.148:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:13.348574 18751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:13.349946 18751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:13.362068 18758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:13.362656 18757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:13.363276 18751 server_base.cc:1034] running on GCE node
W20260927 10:52:13.364084 18760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:13.364632 18751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:13.365446 18751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:13.366647 18751 hybrid_clock.cc:648] HybridClock initialized: now 1790506333366615 us; error 48 us; skew 500 ppm
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:13.370048 18751 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:52:13.371163 18751 webserver.cc:460] Webserver started at http://127.6.31.130:34605/ using document root <none> and password file <none>
I20260927 10:52:13.371877 18751 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:13.371959 18751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:13.372216 18751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:13.387748 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "0cc300af5a334468b71d53ddacf7c3ce"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "6768a2d190e21da4256423db2f8aa997"
server_key_iv: "b879a625da2b9a316052bf24080c4ae7"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:13.388945 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "0cc300af5a334468b71d53ddacf7c3ce"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "6768a2d190e21da4256423db2f8aa997"
server_key_iv: "b879a625da2b9a316052bf24080c4ae7"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:13.400127 18751 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.000s sys 0.004s
I20260927 10:52:13.415477 18767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:13.417273 18751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:52:13.417501 18751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "0cc300af5a334468b71d53ddacf7c3ce"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "6768a2d190e21da4256423db2f8aa997"
server_key_iv: "b879a625da2b9a316052bf24080c4ae7"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:13.417725 18751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:13.447341 18751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:13.450815 18751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:13.451030 18751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:13.451713 18751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:13.452651 18751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:13.452718 18751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:13.452802 18751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:13.452855 18751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:13.463778 18751 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:46467
I20260927 10:52:13.464823 18751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:52:13.465349 18880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:46467 every 8 connection(s)
I20260927 10:52:13.470033 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 18751
I20260927 10:52:13.470175 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:52:13.470490 6270 external_mini_cluster.cc:1428] Setting key 4d4288fbbac8378e0f4e09f105a083bd
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:52:13.484680 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:13.467614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46479 (local address 127.6.31.130:47957)
0927 10:52:13.468732 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:13.468788 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:13.469911 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:13.470781 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:13.470789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:13.471235 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:13.471898 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:13.471917 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.472985 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.472991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:13.474271 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:13.474280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.475184 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.475190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:13.475414 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:13.476153 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:13.476178 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:13.478581 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:13.481213 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:13.481221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:13.481241 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:13.481599 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:13.481958 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:13.481963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:13.481968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:13.482168 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:13.482700 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:13.482706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:13.483098 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 10:52:13.483302 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 10:52:13.483643 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":144,"threads_started":1}
I20260927 10:52:13.485488 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:13.467297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:47957 (local address 127.6.31.190:46479)
0927 10:52:13.467438 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:13.467443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:13.468895 (+ 1452us) server_negotiation.cc:407] Connection header received
0927 10:52:13.470230 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:13.470235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:13.470310 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:13.470376 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:13.472057 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.472820 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.474408 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.474822 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.474947 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:13.478848 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:13.478853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:13.478859 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:13.478966 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:13.481065 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:13.481729 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:13.481734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:13.481737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:13.481841 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:13.482286 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:13.482290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:13.482292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:13.482541 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:13.482643 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:13.485147 (+ 2504us) server_negotiation.cc:299] Negotiation successful
0927 10:52:13.485286 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:52:13.486693 18881 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46479
I20260927 10:52:13.486999 18881 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
I20260927 10:52:13.487659 18881 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
I20260927 10:52:13.488801 18524 ts_manager.cc:194] Registered new tserver with Master: 0cc300af5a334468b71d53ddacf7c3ce (127.6.31.130:46467)
I20260927 10:52:13.489575 18524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:47957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:52:13.514343 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46479
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:13.677325 18886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:13.677716 18886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:13.677827 18886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:13.684248 18886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:13.684443 18886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:13.684588 18886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:52:13.692719 18886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.31.148:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:13.694303 18886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:13.695647 18886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
W20260927 10:52:13.717046 18894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:13.717238 18892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:13.717537 18886 server_base.cc:1034] running on GCE node
W20260927 10:52:13.719162 18891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:13.719740 18886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:13.720525 18886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:13.729411 18886 hybrid_clock.cc:648] HybridClock initialized: now 1790506333729327 us; error 94 us; skew 500 ppm
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:13.733317 18886 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:52:13.734637 18886 webserver.cc:460] Webserver started at http://127.6.31.131:39807/ using document root <none> and password file <none>
I20260927 10:52:13.735426 18886 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:13.735566 18886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:13.735841 18886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:13.753988 18886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "e4410165b33b4c5cb13bed0118e3fa28"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "c70a54df8f6f1b55d4add9c7d7218427"
server_key_iv: "9592ac061a91f15c81b4b04ad3dfd71b"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:13.754714 18886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "e4410165b33b4c5cb13bed0118e3fa28"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "c70a54df8f6f1b55d4add9c7d7218427"
server_key_iv: "9592ac061a91f15c81b4b04ad3dfd71b"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:13.788282 18886 fs_manager.cc:691] Time spent creating directory manager: real 0.033s user 0.007s sys 0.001s
I20260927 10:52:13.814442 18901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:13.818286 18886 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260927 10:52:13.818528 18886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "e4410165b33b4c5cb13bed0118e3fa28"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "c70a54df8f6f1b55d4add9c7d7218427"
server_key_iv: "9592ac061a91f15c81b4b04ad3dfd71b"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:13.818766 18886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:13.852064 18886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:13.856081 18886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:13.856338 18886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:13.857122 18886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:13.858282 18886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:13.858394 18886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:13.858508 18886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:13.858570 18886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:13.872227 18886 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:36787
I20260927 10:52:13.873479 18886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:52:13.874704 19014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:36787 every 8 connection(s)
I20260927 10:52:13.878667 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 18886
I20260927 10:52:13.878793 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:52:13.879102 6270 external_mini_cluster.cc:1428] Setting key ed207ef5a545317ffe87f3edfd0bae0d
Sep 27 10:52:13 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506333, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:13.895674 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:13.879483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:58851 (local address 127.6.31.190:46479)
0927 10:52:13.879680 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:13.879685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:13.881934 (+ 2249us) server_negotiation.cc:407] Connection header received
0927 10:52:13.883291 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:13.883295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:13.883347 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:13.883401 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:13.885114 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.885954 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.887703 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.888053 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.888125 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:13.891235 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:13.891239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:13.891241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:13.891267 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:13.893520 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:13.894016 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:13.894020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:13.894022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:13.894101 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:13.894486 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:13.894490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:13.894492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:13.894679 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:13.894785 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:13.895398 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 10:52:13.895517 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 10:52:13.896660 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:13.877325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46479 (local address 127.6.31.131:58851)
0927 10:52:13.881776 (+ 4451us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:13.881841 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:13.882934 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:13.883828 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:13.883837 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:13.884188 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:13.884947 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:13.884965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.886116 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.886120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:13.887556 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:13.887566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:13.888327 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:13.888331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:13.888521 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:13.889248 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:13.889268 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:13.890998 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:13.893643 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:13.893648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:13.893660 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:13.893905 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:13.894206 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:13.894210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:13.894212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:13.894367 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:13.894796 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:13.894805 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:13.895057 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 10:52:13.895343 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 10:52:13.895672 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4199,"thread_start_us":142,"threads_started":1}
I20260927 10:52:13.897732 19015 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46479
I20260927 10:52:13.897962 19015 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:13.898470 19015 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
I20260927 10:52:13.900051 18524 ts_manager.cc:194] Registered new tserver with Master: e4410165b33b4c5cb13bed0118e3fa28 (127.6.31.131:36787)
I20260927 10:52:13.900620 18524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:58851
I20260927 10:52:13.904819 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:52:13.905535 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 18612
I20260927 10:52:13.915854 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 18751
I20260927 10:52:13.924909 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 18886
I20260927 10:52:13.933210 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 18499
I20260927 10:52:13.940344 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46479
--webserver_interface=127.6.31.190
--webserver_port=39179
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:46479
--ranger_config_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:45363/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.31.148:45363/kms with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:14.058080 19018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:14.058369 19018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:14.058409 19018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:14.062597 19018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:14.062670 19018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:14.062690 19018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:14.062705 19018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:14.062721 19018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:14.068393 19018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.31.148:45363/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:46479
--ranger_config_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:14.069530 19018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:14.070600 19018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:14.079607 19026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:14.079596 19023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:14.079577 19024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:14.080148 19018 server_base.cc:1034] running on GCE node
I20260927 10:52:14.080768 19018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:14.081965 19018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:14.083166 19018 hybrid_clock.cc:648] HybridClock initialized: now 1790506334083142 us; error 39 us; skew 500 ppm
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:14.086051 19018 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:52:14.087172 19018 webserver.cc:460] Webserver started at http://127.6.31.190:39179/ using document root <none> and password file <none>
I20260927 10:52:14.087726 19018 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:14.087821 19018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:14.142560 19018 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:52:14.144860 19034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.146085 19018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:52:14.146240 19018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "dde4daf24ed245dbb23620eb5f3b7de9"
format_stamp: "Formatted at 2026-09-27 10:52:11 on dist-test-slave-c60x"
server_key: "c84a7047e105335aad17f7f3bf2c933d"
server_key_iv: "3146df4d4d850ae0b93ee9af7af731ae"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:14.146792 19018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:14.159662 19018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:14.162557 19018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:14.162793 19018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:14.170325 19018 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:46479
I20260927 10:52:14.170353 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:46479 every 8 connection(s)
I20260927 10:52:14.171482 19018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:52:14.176029 19087 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:52:14.177757 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Bootstrap starting.
I20260927 10:52:14.178241 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 19018
I20260927 10:52:14.178789 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:36283
--local_ip_for_outbound_sockets=127.6.31.129
--tserver_master_addrs=127.6.31.190:46479
--webserver_port=40635
--webserver_interface=127.6.31.129
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.31.148:45363/kms with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:52:14.194952 19087 log.cc:824] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:14.201596 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:52:14.202025 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Bootstrap complete.
I20260927 10:52:14.206234 19087 raft_consensus.cc:348] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } }
I20260927 10:52:14.206599 19087 raft_consensus.cc:729] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde4daf24ed245dbb23620eb5f3b7de9, State: Initialized, Role: FOLLOWER
I20260927 10:52:14.207208 19087 consensus_queue.cc:260] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } }
I20260927 10:52:14.207325 19087 raft_consensus.cc:388] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:52:14.207404 19087 raft_consensus.cc:482] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:52:14.207496 19087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:52:14.209496 19087 raft_consensus.cc:504] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } }
I20260927 10:52:14.209820 19087 leader_election.cc:302] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dde4daf24ed245dbb23620eb5f3b7de9; no voters:
I20260927 10:52:14.210144 19087 leader_election.cc:288] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:52:14.210235 19091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:52:14.210482 19091 raft_consensus.cc:686] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [term 2 LEADER]: Becoming Leader. State: Replica: dde4daf24ed245dbb23620eb5f3b7de9, State: Running, Role: LEADER
I20260927 10:52:14.210816 19091 consensus_queue.cc:237] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } }
I20260927 10:52:14.211367 19087 sys_catalog.cc:564] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:52:14.212293 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dde4daf24ed245dbb23620eb5f3b7de9. Latest consensus state: current_term: 2 leader_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } } }
I20260927 10:52:14.212338 19092 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4daf24ed245dbb23620eb5f3b7de9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46479 } } }
I20260927 10:52:14.212426 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:14.212450 19092 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:14.213785 19087 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:52:14.213935 19100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:52:14.219578 19100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:52:14.220309 19100 catalog_manager.cc:1259] Loaded cluster ID: 34ad72aaf08e4579b3e15c7189058735
I20260927 10:52:14.220427 19100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:52:14.222968 19100 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:52:14.223724 19100 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Loaded TSK: 0
I20260927 10:52:14.224995 19100 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:52:14.316608 19089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:14.317065 19089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:14.317212 19089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:14.323168 19089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:14.323321 19089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:14.323477 19089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:52:14.329262 19089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.31.148:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:36283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=40635
--tserver_master_addrs=127.6.31.190:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:14.330327 19089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:14.331423 19089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:14.341517 19124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:14.341564 19127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:14.341771 19125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:14.342291 19089 server_base.cc:1034] running on GCE node
I20260927 10:52:14.342692 19089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:14.343447 19089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:14.344671 19089 hybrid_clock.cc:648] HybridClock initialized: now 1790506334344597 us; error 39 us; skew 500 ppm
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:14.348078 19089 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:52:14.349225 19089 webserver.cc:460] Webserver started at http://127.6.31.129:40635/ using document root <none> and password file <none>
I20260927 10:52:14.349876 19089 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:14.349980 19089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:14.379900 19089 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.002s sys 0.000s
I20260927 10:52:14.383971 19135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.385599 19089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 10:52:14.385741 19089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "cf545a05264d4438964dd9e0bfd27319"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "15337537425d4462bbfbe83cc899528e"
server_key_iv: "c40ca923b6dd877bca4f9d6477ba3512"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:14.386463 19089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:14.415242 19089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:14.418977 19089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:14.419255 19089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:14.420010 19089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:14.421044 19089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:14.421111 19089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.421166 19089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:14.421202 19089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.437263 19089 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:36283
I20260927 10:52:14.438603 19089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:52:14.439548 19248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:36283 every 8 connection(s)
I20260927 10:52:14.442885 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 19089
I20260927 10:52:14.443423 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:46467
--local_ip_for_outbound_sockets=127.6.31.130
--tserver_master_addrs=127.6.31.190:46479
--webserver_port=34605
--webserver_interface=127.6.31.130
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.31.148:45363/kms with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:14.466631 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:14.441106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:56313 (local address 127.6.31.190:46479)
0927 10:52:14.441722 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:14.441745 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:14.442201 (+ 456us) server_negotiation.cc:407] Connection header received
0927 10:52:14.443969 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:14.444271 (+ 302us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:14.446939 (+ 2668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:14.447332 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:14.449546 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.450682 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.452384 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.452942 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.453086 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:14.458691 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:14.458698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:14.458726 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:14.458761 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:14.461062 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:14.462672 (+ 1610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:14.462679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:14.462685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:14.462905 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:14.463532 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:14.463536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:14.463538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:14.464029 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:14.464157 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:14.465232 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0927 10:52:14.465568 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":125,"threads_started":1}
I20260927 10:52:14.468333 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:14.441239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46479 (local address 127.6.31.129:56313)
0927 10:52:14.442010 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:14.442060 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:14.443188 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:14.448560 (+ 5372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:14.448566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:14.448931 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:14.449375 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:14.449388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.450840 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.450843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:14.452243 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:14.452254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.453344 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.453347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:14.453474 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:14.454111 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:14.454134 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:14.458452 (+ 4318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:14.461487 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:14.461493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:14.461508 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:14.461854 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:14.463229 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:14.463232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:14.463234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:14.463415 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:14.464556 (+ 1141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:14.464562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:14.464869 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 10:52:14.467136 (+ 2267us) client_negotiation.cc:240] Negotiation successful
0927 10:52:14.467468 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":132,"threads_started":1}
I20260927 10:52:14.471150 19249 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46479
I20260927 10:52:14.471411 19249 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:14.471983 19249 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
W20260927 10:52:14.474188 19249 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:46479 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:52:14.474313 19249 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:14.474489 19249 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
I20260927 10:52:14.475328 19249 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:14.475498 19249 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
W20260927 10:52:14.476193 19249 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:52:14.581561 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:14.581877 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:14.581956 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:14.586215 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:14.586310 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:14.586411 19253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:52:14.593570 19253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.31.148:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:46467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=34605
--tserver_master_addrs=127.6.31.190:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:14.594663 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:14.595867 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:14.606734 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:14.606750 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:14.607538 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:14.608233 19253 server_base.cc:1034] running on GCE node
I20260927 10:52:14.608688 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:14.609474 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:14.610688 19253 hybrid_clock.cc:648] HybridClock initialized: now 1790506334610666 us; error 33 us; skew 500 ppm
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:14.614495 19253 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:52:14.615665 19253 webserver.cc:460] Webserver started at http://127.6.31.130:34605/ using document root <none> and password file <none>
I20260927 10:52:14.616415 19253 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:14.616523 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:14.641286 19253 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260927 10:52:14.644831 19269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.646302 19253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:52:14.646452 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "0cc300af5a334468b71d53ddacf7c3ce"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "6768a2d190e21da4256423db2f8aa997"
server_key_iv: "b879a625da2b9a316052bf24080c4ae7"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:14.647046 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:14.666996 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:14.672048 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:14.672273 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:14.672974 19253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:14.674072 19253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:14.674140 19253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.674196 19253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:14.674230 19253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.687821 19253 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:46467
I20260927 10:52:14.687870 19382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:46467 every 8 connection(s)
I20260927 10:52:14.689182 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:52:14.695371 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 19253
I20260927 10:52:14.695863 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.31.148:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:36787
--local_ip_for_outbound_sockets=127.6.31.131
--tserver_master_addrs=127.6.31.190:46479
--webserver_port=39807
--webserver_interface=127.6.31.131
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.31.148:45363/kms with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:14.716605 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:14.697547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:49331 (local address 127.6.31.190:46479)
0927 10:52:14.697679 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:14.697686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:14.697702 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:52:14.697765 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:14.697770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:14.697838 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:14.697929 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:14.705083 (+ 7154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.705939 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.707690 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.708038 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.708113 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:14.711283 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:14.711286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:14.711289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:14.711317 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:14.713581 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:14.714456 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:14.714462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:14.714465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:14.714559 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:14.715001 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:14.715007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:14.715011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:14.715314 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:14.715414 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:14.716168 (+ 754us) server_negotiation.cc:299] Negotiation successful
0927 10:52:14.716339 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:52:14.718708 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:14.694069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46479 (local address 127.6.31.130:49331)
0927 10:52:14.694762 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:14.694820 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:14.695899 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:14.699730 (+ 3831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:14.699740 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:14.704040 (+ 4300us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:14.704853 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:14.704878 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.706109 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.706113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:14.707545 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:14.707556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.708336 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.708339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:14.708533 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:14.709271 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:14.709298 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:14.711049 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:14.713737 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:14.713745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:14.713766 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:14.714324 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:14.714677 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:14.714681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:14.714685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:14.714874 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:14.715491 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:14.715501 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:14.715855 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 10:52:14.717125 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0927 10:52:14.717515 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":155,"threads_started":1}
I20260927 10:52:14.720364 19383 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46479
I20260927 10:52:14.720629 19383 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:14.721206 19383 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
W20260927 10:52:14.722860 19383 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:46479 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:52:14.722992 19383 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:14.723203 19383 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
I20260927 10:52:14.724020 19383 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:14.724191 19383 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
W20260927 10:52:14.725140 19383 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:52:14.846028 19386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:14.846431 19386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:14.846555 19386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:14.852355 19386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:14.852435 19386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:14.852524 19386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:52:14.860558 19386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.31.148:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:36787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=39807
--tserver_master_addrs=127.6.31.190:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:14.861924 19386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:14.863250 19386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:14.875567 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:14.875567 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:14.875799 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:14.877390 19386 server_base.cc:1034] running on GCE node
I20260927 10:52:14.877908 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:14.878677 19386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:14.879858 19386 hybrid_clock.cc:648] HybridClock initialized: now 1790506334879842 us; error 30 us; skew 500 ppm
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:14.882887 19386 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:52:14.883945 19386 webserver.cc:460] Webserver started at http://127.6.31.131:39807/ using document root <none> and password file <none>
I20260927 10:52:14.884671 19386 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:14.884776 19386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:14.907680 19386 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260927 10:52:14.910894 19403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.912537 19386 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:52:14.912784 19386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "e4410165b33b4c5cb13bed0118e3fa28"
format_stamp: "Formatted at 2026-09-27 10:52:13 on dist-test-slave-c60x"
server_key: "c70a54df8f6f1b55d4add9c7d7218427"
server_key_iv: "9592ac061a91f15c81b4b04ad3dfd71b"
server_key_version: "kuduclusterkey@0"
I20260927 10:52:14.913378 19386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:14.929435 19386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:14.932910 19386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:14.933133 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:14.933884 19386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:14.935184 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:14.935249 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.935487 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:14.935559 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:14.948661 19386 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:36787
I20260927 10:52:14.948827 19516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:36787 every 8 connection(s)
I20260927 10:52:14.950030 19386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:52:14.956854 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 19386
Sep 27 10:52:14 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506334, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:14.970037 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:14.957647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.6.31.190:46479)
0927 10:52:14.959714 (+ 2067us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:14.959718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:14.959729 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:52:14.959763 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:14.959765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:14.959803 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:14.959931 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:14.960963 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.961832 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.962946 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.963364 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.963486 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:14.965383 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:14.965388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:14.965391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:14.965421 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:14.968063 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:14.968590 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:14.968593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:14.968595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:14.968699 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:14.969038 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:14.969041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:14.969043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:14.969270 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:14.969477 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:14.969712 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 10:52:14.969847 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1995,"thread_start_us":101,"threads_started":1}
I20260927 10:52:14.973392 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:14.954091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:38505 (local address 127.6.31.190:46479)
0927 10:52:14.954233 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:14.954238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:14.954256 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:52:14.954831 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:14.954835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:14.954902 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:14.955012 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:14.959074 (+ 4062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.960009 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.961683 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.961982 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.962101 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:14.965564 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:14.965570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:14.965573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:14.965626 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:14.967703 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:14.971365 (+ 3662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:14.971368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:14.971370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:14.971458 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:14.972007 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:14.972011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:14.972013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:14.972252 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:14.972362 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:14.973079 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 10:52:14.973223 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:52:14.974493 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:14.952961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:46479 (local address 127.6.31.131:38505)
0927 10:52:14.953502 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:14.953560 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:14.954605 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:14.958050 (+ 3445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:14.958059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:14.958367 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:14.958899 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:14.958915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.960172 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.960177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:14.961547 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:14.961557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:14.962168 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:14.962173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:14.962414 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:14.963379 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:14.963397 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:14.965349 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:14.967839 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:14.967845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:14.967863 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:14.968213 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:14.971588 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:14.971591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:14.971593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:14.971841 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:14.972386 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:14.972394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:14.972762 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 10:52:14.972981 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 10:52:14.973352 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":166,"threads_started":1}
I20260927 10:52:14.975816 19517 heartbeater.cc:347] Connected to a master server at 127.6.31.190:46479
I20260927 10:52:14.976087 19517 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:14.976756 19517 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
W20260927 10:52:14.978528 19517 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:46479 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:52:14.978667 19517 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:14.978866 19517 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
I20260927 10:52:14.979857 19517 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:14.980057 19517 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
W20260927 10:52:14.980978 19517 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:52:15 dist-test-slave-c60x krb5kdc[18055](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506335, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:15.476436 19249 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:15.476756 19249 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
I20260927 10:52:15.478052 19051 ts_manager.cc:194] Registered new tserver with Master: cf545a05264d4438964dd9e0bfd27319 (127.6.31.129:36283)
I20260927 10:52:15.479905 19051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:56313
I20260927 10:52:15.725378 19383 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:15.725690 19383 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
I20260927 10:52:15.726461 19051 ts_manager.cc:194] Registered new tserver with Master: 0cc300af5a334468b71d53ddacf7c3ce (127.6.31.130:46467)
I20260927 10:52:15.727253 19051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:49331
Sep 27 10:52:15 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506335, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:15.981207 19517 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:15.981525 19517 heartbeater.cc:511] Master 127.6.31.190:46479 requested a full tablet report, sending...
I20260927 10:52:15.982239 19051 ts_manager.cc:194] Registered new tserver with Master: e4410165b33b4c5cb13bed0118e3fa28 (127.6.31.131:36787)
I20260927 10:52:15.983012 19051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:38505
I20260927 10:52:15.993074 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:52:16.003957 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:15.994771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.6.31.190:46479)
0927 10:52:15.995024 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:15.995029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:15.995387 (+ 358us) server_negotiation.cc:407] Connection header received
0927 10:52:15.995549 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:15.995553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:15.995601 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:15.995711 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:15.996495 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:15.997275 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:15.998084 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:15.998329 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:15.998442 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:15.999908 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:15.999912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:15.999915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:15.999946 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:16.001870 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:16.002359 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:16.002363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:16.002365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:16.002466 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:16.002819 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:16.002823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:16.002825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:16.003052 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:16.003217 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:16.003620 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 10:52:16.003771 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
Sep 27 10:52:16 dist-test-slave-c60x krb5kdc[18055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506335, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for HTTP/127.6.31.148@KRBTEST.COM
I20260927 10:52:16.011351 19051 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:52:16.024603 19051 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:52:16.086150 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.076121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.6.31.129:36283)
0927 10:52:16.076396 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:16.076403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:16.078212 (+ 1809us) server_negotiation.cc:407] Connection header received
0927 10:52:16.078501 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:16.078507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:16.078722 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:16.078846 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:16.079957 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.080953 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.083768 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.084620 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.084713 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.084888 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:16.085823 (+ 935us) server_negotiation.cc:299] Negotiation successful
0927 10:52:16.085931 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":99,"threads_started":1}
I20260927 10:52:16.086855 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.075289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36283 (local address 127.0.0.1:42606)
0927 10:52:16.078050 (+ 2761us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:16.078110 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:16.078326 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:16.079651 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:16.079657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:16.079683 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:16.079817 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.079825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.081586 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.081589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.083624 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.083635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.085486 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.085490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.085597 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.085637 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:52:16.086621 (+ 984us) client_negotiation.cc:240] Negotiation successful
0927 10:52:16.086702 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2471,"thread_start_us":130,"threads_started":1}
I20260927 10:52:16.087033 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.077180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.6.31.131:36787)
0927 10:52:16.077468 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:16.077474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:16.081826 (+ 4352us) server_negotiation.cc:407] Connection header received
0927 10:52:16.081907 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:16.081912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:16.082084 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:16.082211 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:16.082569 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.083811 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.085240 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.086042 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.086144 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.086272 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:16.086780 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 10:52:16.086878 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
I20260927 10:52:16.087182 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.076780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:36787 (local address 127.0.0.1:37280)
0927 10:52:16.081631 (+ 4851us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:16.081645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:16.081727 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:16.082346 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:16.082349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:16.082365 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:16.082448 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.082453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.083939 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.083942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.085098 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.085108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.086331 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.086334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.086417 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.086439 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:52:16.086510 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:52:16.086657 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4778,"thread_start_us":101,"threads_started":1}
I20260927 10:52:16.090682 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.075410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:46467 (local address 127.0.0.1:40338)
0927 10:52:16.078988 (+ 3578us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:16.079003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:16.079114 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:16.079988 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:16.079993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:16.080012 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:16.080101 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.080106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.082490 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.082494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.084670 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.084682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.089677 (+ 4995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.089681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.089776 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.089824 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 10:52:16.089891 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:52:16.090559 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3511,"thread_start_us":77,"threads_started":1}
I20260927 10:52:16.090566 19183 tablet_service.cc:1467] Processing CreateTablet for tablet 87ab93ef20c346f08537579dd085b585 (DEFAULT_TABLE table=test-table [id=68b6071a58c743b4ab2662b00a519d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:16.091131 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.075575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.6.31.130:46467)
0927 10:52:16.079380 (+ 3805us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:16.079386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:16.079408 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:52:16.079484 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:16.079491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:16.079690 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:16.079833 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:16.080229 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.081393 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.084804 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.089447 (+ 4643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.090022 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.090268 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:16.090324 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 10:52:16.090414 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3693,"thread_start_us":104,"threads_started":1}
I20260927 10:52:16.094095 19451 tablet_service.cc:1467] Processing CreateTablet for tablet 87ab93ef20c346f08537579dd085b585 (DEFAULT_TABLE table=test-table [id=68b6071a58c743b4ab2662b00a519d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:16.095276 19451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ab93ef20c346f08537579dd085b585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:16.097143 19317 tablet_service.cc:1467] Processing CreateTablet for tablet 87ab93ef20c346f08537579dd085b585 (DEFAULT_TABLE table=test-table [id=68b6071a58c743b4ab2662b00a519d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:16.098133 19317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ab93ef20c346f08537579dd085b585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:16.104177 19552 tablet_bootstrap.cc:492] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: Bootstrap starting.
I20260927 10:52:16.106515 19183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ab93ef20c346f08537579dd085b585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:16.106933 19552 tablet_bootstrap.cc:654] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:16.107824 19552 log.cc:824] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:16.110140 19552 tablet_bootstrap.cc:492] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: No bootstrap required, opened a new log
I20260927 10:52:16.110363 19552 ts_tablet_manager.cc:1397] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:52:16.111572 19553 tablet_bootstrap.cc:492] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: Bootstrap starting.
I20260927 10:52:16.114880 19552 raft_consensus.cc:348] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } }
I20260927 10:52:16.115334 19552 raft_consensus.cc:374] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:16.115494 19552 raft_consensus.cc:729] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4410165b33b4c5cb13bed0118e3fa28, State: Initialized, Role: FOLLOWER
I20260927 10:52:16.116091 19552 consensus_queue.cc:260] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } }
I20260927 10:52:16.117349 19552 ts_tablet_manager.cc:1428] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:52:16.117758 19517 heartbeater.cc:503] Master 127.6.31.190:46479 was elected leader, sending a full tablet report...
I20260927 10:52:16.117990 19556 tablet_bootstrap.cc:492] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319: Bootstrap starting.
I20260927 10:52:16.121953 19556 tablet_bootstrap.cc:654] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:16.123675 19556 log.cc:824] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:16.125886 19556 tablet_bootstrap.cc:492] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319: No bootstrap required, opened a new log
I20260927 10:52:16.126173 19556 ts_tablet_manager.cc:1397] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 10:52:16.126830 19553 tablet_bootstrap.cc:654] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:16.127851 19553 log.cc:824] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:16.130389 19553 tablet_bootstrap.cc:492] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: No bootstrap required, opened a new log
I20260927 10:52:16.130600 19553 ts_tablet_manager.cc:1397] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260927 10:52:16.130676 19556 raft_consensus.cc:348] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } }
I20260927 10:52:16.130980 19556 raft_consensus.cc:374] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:16.131040 19556 raft_consensus.cc:729] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf545a05264d4438964dd9e0bfd27319, State: Initialized, Role: FOLLOWER
I20260927 10:52:16.131606 19556 consensus_queue.cc:260] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } }
I20260927 10:52:16.132717 19556 ts_tablet_manager.cc:1428] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:52:16.133124 19249 heartbeater.cc:503] Master 127.6.31.190:46479 was elected leader, sending a full tablet report...
I20260927 10:52:16.137691 19553 raft_consensus.cc:348] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } }
I20260927 10:52:16.137971 19553 raft_consensus.cc:374] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:16.138047 19553 raft_consensus.cc:729] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc300af5a334468b71d53ddacf7c3ce, State: Initialized, Role: FOLLOWER
I20260927 10:52:16.138543 19553 consensus_queue.cc:260] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } }
I20260927 10:52:16.139245 19383 heartbeater.cc:503] Master 127.6.31.190:46479 was elected leader, sending a full tablet report...
I20260927 10:52:16.139701 19553 ts_tablet_manager.cc:1428] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 10:52:16.175905 19560 raft_consensus.cc:482] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:16.176156 19560 raft_consensus.cc:504] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } }
I20260927 10:52:16.177671 19560 leader_election.cc:288] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf545a05264d4438964dd9e0bfd27319 (127.6.31.129:36283), e4410165b33b4c5cb13bed0118e3fa28 (127.6.31.131:36787)
W20260927 10:52:16.193581 19250 tablet.cc:2369] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:52:16.194418 19384 tablet.cc:2369] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:52:16.199553 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.179560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:60571 (local address 127.6.31.131:36787)
0927 10:52:16.179676 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:16.179683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:16.188700 (+ 9017us) server_negotiation.cc:407] Connection header received
0927 10:52:16.189079 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:16.189083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:16.189142 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:16.189192 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:16.194643 (+ 5451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.195443 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.196981 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.197595 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.197718 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.197777 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:16.199307 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0927 10:52:16.199406 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 10:52:16.200018 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.178845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:36787 (local address 127.6.31.130:60571)
0927 10:52:16.188587 (+ 9742us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:16.188602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:16.188805 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:16.189612 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:16.189615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:16.189633 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:16.189881 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.189889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.195597 (+ 5708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.195600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.196823 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.196835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.197883 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.197886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.197987 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.198012 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:52:16.199746 (+ 1734us) client_negotiation.cc:240] Negotiation successful
0927 10:52:16.199837 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9673,"thread_start_us":85,"threads_started":1}
I20260927 10:52:16.200446 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.179401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:60371 (local address 127.6.31.129:36283)
0927 10:52:16.179560 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:16.179566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:16.188368 (+ 8802us) server_negotiation.cc:407] Connection header received
0927 10:52:16.189357 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:16.189361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:16.189417 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:16.189469 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:16.192111 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.192914 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.198153 (+ 5239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.198766 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.198869 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.198919 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:16.200234 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0927 10:52:16.200326 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:52:16.200819 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.178405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36283 (local address 127.6.31.130:60371)
0927 10:52:16.188225 (+ 9820us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:16.188249 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:16.188928 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:16.190034 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:16.190037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:16.190055 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:16.190179 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.190185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.193075 (+ 2890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.193078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.194470 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:16.194482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.199019 (+ 4537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.199022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:16.199150 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.199173 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:52:16.200585 (+ 1412us) client_negotiation.cc:240] Negotiation successful
0927 10:52:16.200677 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9708,"thread_start_us":121,"threads_started":1}
I20260927 10:52:16.200837 19470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ab93ef20c346f08537579dd085b585" candidate_uuid: "0cc300af5a334468b71d53ddacf7c3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4410165b33b4c5cb13bed0118e3fa28" is_pre_election: true
I20260927 10:52:16.201233 19470 raft_consensus.cc:2454] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cc300af5a334468b71d53ddacf7c3ce in term 0.
I20260927 10:52:16.201748 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ab93ef20c346f08537579dd085b585" candidate_uuid: "0cc300af5a334468b71d53ddacf7c3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf545a05264d4438964dd9e0bfd27319" is_pre_election: true
I20260927 10:52:16.202000 19203 raft_consensus.cc:2454] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cc300af5a334468b71d53ddacf7c3ce in term 0.
W20260927 10:52:16.202283 19518 tablet.cc:2369] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:52:16.202540 19272 leader_election.cc:302] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc300af5a334468b71d53ddacf7c3ce, e4410165b33b4c5cb13bed0118e3fa28; no voters:
I20260927 10:52:16.202876 19560 raft_consensus.cc:2781] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:52:16.202960 19560 raft_consensus.cc:482] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:52:16.203008 19560 raft_consensus.cc:3037] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:16.204457 19560 raft_consensus.cc:504] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } }
I20260927 10:52:16.204862 19560 leader_election.cc:288] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [CANDIDATE]: Term 1 election: Requested vote from peers cf545a05264d4438964dd9e0bfd27319 (127.6.31.129:36283), e4410165b33b4c5cb13bed0118e3fa28 (127.6.31.131:36787)
I20260927 10:52:16.205677 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ab93ef20c346f08537579dd085b585" candidate_uuid: "0cc300af5a334468b71d53ddacf7c3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf545a05264d4438964dd9e0bfd27319"
I20260927 10:52:16.205677 19470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ab93ef20c346f08537579dd085b585" candidate_uuid: "0cc300af5a334468b71d53ddacf7c3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4410165b33b4c5cb13bed0118e3fa28"
I20260927 10:52:16.205811 19470 raft_consensus.cc:3037] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:16.205811 19203 raft_consensus.cc:3037] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:16.207149 19203 raft_consensus.cc:2454] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cc300af5a334468b71d53ddacf7c3ce in term 1.
I20260927 10:52:16.207147 19470 raft_consensus.cc:2454] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cc300af5a334468b71d53ddacf7c3ce in term 1.
I20260927 10:52:16.207603 19270 leader_election.cc:302] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc300af5a334468b71d53ddacf7c3ce, cf545a05264d4438964dd9e0bfd27319; no voters:
I20260927 10:52:16.207988 19560 raft_consensus.cc:2781] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:52:16.208273 19560 raft_consensus.cc:686] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 1 LEADER]: Becoming Leader. State: Replica: 0cc300af5a334468b71d53ddacf7c3ce, State: Running, Role: LEADER
I20260927 10:52:16.208642 19560 consensus_queue.cc:237] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } }
I20260927 10:52:16.212122 19051 catalog_manager.cc:5495] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cc300af5a334468b71d53ddacf7c3ce (127.6.31.130). New cstate: current_term: 1 leader_uuid: "0cc300af5a334468b71d53ddacf7c3ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cc300af5a334468b71d53ddacf7c3ce" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 } health_report { overall_health: UNKNOWN } } }
I20260927 10:52:16.244587 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.239170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.6.31.130:46467)
0927 10:52:16.239361 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:16.239366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:16.239380 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:52:16.239469 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:16.239472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:16.239545 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:16.239657 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:52:16.240088 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.240895 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.242219 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:16.242560 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:16.242679 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:16.243164 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:52:16.243768 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:52:16.243889 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:16.244347 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 10:52:16.244414 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 10:52:16.252833 19470 raft_consensus.cc:1261] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [term 1 FOLLOWER]: Refusing update from remote peer 0cc300af5a334468b71d53ddacf7c3ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:16.253834 19563 consensus_queue.cc:1035] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4410165b33b4c5cb13bed0118e3fa28" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 36787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:16.259516 19203 raft_consensus.cc:1261] T 87ab93ef20c346f08537579dd085b585 P cf545a05264d4438964dd9e0bfd27319 [term 1 FOLLOWER]: Refusing update from remote peer 0cc300af5a334468b71d53ddacf7c3ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:16.260361 19563 consensus_queue.cc:1035] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf545a05264d4438964dd9e0bfd27319" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:16.266208 19566 mvcc.cc:205] Tried to move back new op lower bound from 7333913953281163264 to 7333913953113714688. Current Snapshot: MvccSnapshot[applied={T|T < 7333913953281163264}]
I20260927 10:52:16.268976 19567 mvcc.cc:205] Tried to move back new op lower bound from 7333913953281163264 to 7333913953113714688. Current Snapshot: MvccSnapshot[applied={T|T < 7333913953281163264}]
I20260927 10:52:16.279353 19569 mvcc.cc:205] Tried to move back new op lower bound from 7333913953281163264 to 7333913953113714688. Current Snapshot: MvccSnapshot[applied={T|T < 7333913953281163264 or (T in {7333913953281163264})}]
I20260927 10:52:16.279841 19051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:52:16.280043 19051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:52:16.282420 19051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dde4daf24ed245dbb23620eb5f3b7de9: Sending DeleteTablet for 3 replicas of tablet 87ab93ef20c346f08537579dd085b585
I20260927 10:52:16.283224 19183 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ab93ef20c346f08537579dd085b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:16 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:42606
I20260927 10:52:16.283643 19317 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ab93ef20c346f08537579dd085b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:16 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:40338
I20260927 10:52:16.284785 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 19089
I20260927 10:52:16.284754 19451 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ab93ef20c346f08537579dd085b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:16 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:37280
I20260927 10:52:16.291425 19576 tablet_replica.cc:330] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: stopping tablet replica
I20260927 10:52:16.292006 19576 raft_consensus.cc:2229] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:52:16.292532 19576 raft_consensus.cc:2258] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:52:16.294396 19576 ts_tablet_manager.cc:1905] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:52:16.299345 19577 tablet_replica.cc:330] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: stopping tablet replica
I20260927 10:52:16.299851 19577 raft_consensus.cc:2229] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:52:16.300303 19577 raft_consensus.cc:2258] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:52:16.302062 19577 ts_tablet_manager.cc:1905] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:52:16.302670 19576 ts_tablet_manager.cc:1918] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:52:16.302765 19576 log.cc:1196] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/87ab93ef20c346f08537579dd085b585
I20260927 10:52:16.303154 19576 ts_tablet_manager.cc:1939] T 87ab93ef20c346f08537579dd085b585 P 0cc300af5a334468b71d53ddacf7c3ce: Deleting consensus metadata
I20260927 10:52:16.304585 19038 catalog_manager.cc:4841] TS 0cc300af5a334468b71d53ddacf7c3ce (127.6.31.130:46467): tablet 87ab93ef20c346f08537579dd085b585 (table test-table [id=68b6071a58c743b4ab2662b00a519d5b]) successfully deleted
W20260927 10:52:16.304768 19035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:36283 (error 108)
I20260927 10:52:16.305173 19577 ts_tablet_manager.cc:1918] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:52:16.305326 19577 log.cc:1196] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/87ab93ef20c346f08537579dd085b585
I20260927 10:52:16.305629 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 19253
I20260927 10:52:16.305757 19577 ts_tablet_manager.cc:1939] T 87ab93ef20c346f08537579dd085b585 P e4410165b33b4c5cb13bed0118e3fa28: Deleting consensus metadata
I20260927 10:52:16.307667 19037 catalog_manager.cc:4841] TS e4410165b33b4c5cb13bed0118e3fa28 (127.6.31.131:36787): tablet 87ab93ef20c346f08537579dd085b585 (table test-table [id=68b6071a58c743b4ab2662b00a519d5b]) successfully deleted
I20260927 10:52:16.308647 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:16.308338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36283 (local address 127.0.0.1:42608)
0927 10:52:16.308461 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:16.308543 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 10:52:16.308926 19035 catalog_manager.cc:4584] TS cf545a05264d4438964dd9e0bfd27319 (127.6.31.129:36283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87ab93ef20c346f08537579dd085b585: Network error: Client connection negotiation failed: client connection to 127.6.31.129:36283: connect: Connection refused (error 111)
I20260927 10:52:16.313694 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 19386
I20260927 10:52:16.320082 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 19018
I20260927 10:52:16.330627 6270 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:52:16.665869 6270 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:52:16.665977 6270 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:52:17.011204 6270 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:52:17.011 UTC [18085] LOG: received smart shutdown request
2026-09-27 10:52:17.015 UTC [18085] LOG: background worker "logical replication launcher" (PID 18094) exited with exit code 1
2026-09-27 10:52:17.015 UTC [18089] LOG: shutting down
2026-09-27 10:52:17.205 UTC [18085] LOG: database system is shut down
2026-09-27T10:52:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48444 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:52:17 dist-test-slave-c60x krb5kdc[19587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:52:17 dist-test-slave-c60x krb5kdc[19587](info): set up 2 sockets
Sep 27 10:52:17 dist-test-slave-c60x krb5kdc[19587](info): commencing operation
krb5kdc: starting...
W20260927 10:52:18.391342 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:52:18 dist-test-slave-c60x krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:52:18.518812 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:39341
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:39341,127.6.31.189:37921,127.6.31.188:41223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:18.628770 19600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:18.629071 19600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:18.629140 19600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:18.633200 19600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:18.633294 19600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:18.633328 19600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:18.633352 19600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:18.633375 19600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:18.638610 19600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:39341,127.6.31.189:37921,127.6.31.188:41223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:39341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:18.639796 19600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:18.640832 19600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:18.649372 19608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:18.649391 19606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:18.649374 19605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:18.649751 19600 server_base.cc:1034] running on GCE node
I20260927 10:52:18.650326 19600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:18.651484 19600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:18.652674 19600 hybrid_clock.cc:648] HybridClock initialized: now 1790506338652641 us; error 49 us; skew 500 ppm
Sep 27 10:52:18 dist-test-slave-c60x krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:18.655527 19600 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:52:18.656353 19600 webserver.cc:460] Webserver started at http://127.6.31.190:46389/ using document root <none> and password file <none>
I20260927 10:52:18.656878 19600 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:18.656936 19600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:18.657106 19600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:18.659183 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "72feaf6673f448daa28c6a3d1d3fd44c"
format_stamp: "Formatted at 2026-09-27 10:52:18 on dist-test-slave-c60x"
server_key: "7a39387e787139fc860a51b8d02ac866"
server_key_iv: "41e7ac439f6401bd3db136cba24cb07c"
server_key_version: "encryptionkey@0"
I20260927 10:52:18.659767 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "72feaf6673f448daa28c6a3d1d3fd44c"
format_stamp: "Formatted at 2026-09-27 10:52:18 on dist-test-slave-c60x"
server_key: "7a39387e787139fc860a51b8d02ac866"
server_key_iv: "41e7ac439f6401bd3db136cba24cb07c"
server_key_version: "encryptionkey@0"
I20260927 10:52:18.663647 19600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 10:52:18.666194 19615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:18.667568 19600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:52:18.667784 19600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "72feaf6673f448daa28c6a3d1d3fd44c"
format_stamp: "Formatted at 2026-09-27 10:52:18 on dist-test-slave-c60x"
server_key: "7a39387e787139fc860a51b8d02ac866"
server_key_iv: "41e7ac439f6401bd3db136cba24cb07c"
server_key_version: "encryptionkey@0"
I20260927 10:52:18.667917 19600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:18.684100 19600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:18.687847 19600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:18.688062 19600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:18.695386 19600 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:39341
I20260927 10:52:18.695408 19667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:39341 every 8 connection(s)
I20260927 10:52:18.696327 19600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:52:18.699203 19668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:18.703459 19668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:52:18.705325 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 19600
I20260927 10:52:18.705510 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:52:18.705780 6270 external_mini_cluster.cc:1428] Setting key 50131254525b13d6ac207b92fa00e24c
WARNING: no policy specified for kudu/127.6.31.189@KRBTEST.COM; defaulting to no policy
Sep 27 10:52:18 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.189@KRBTEST.COM" created.
I20260927 10:52:18.720398 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.706606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.6.31.190:39341)
0927 10:52:18.706878 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:18.706889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:18.706934 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:52:18.707984 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:18.707992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:18.708257 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:18.708460 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:18.709462 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.710450 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.712090 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.712464 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.712589 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:18.715330 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:18.715335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:18.715364 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:18.715400 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:18.717038 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:18.717629 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:18.717636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:18.717638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:18.717782 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:18.718082 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:18.718089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:18.718094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:18.718614 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:18.718753 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:18.719128 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 10:52:18.719373 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260927 10:52:18.720397 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.705816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:39341 (local address 127.0.0.1:39490)
0927 10:52:18.706375 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.706436 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:18.707666 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:18.708462 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:18.708466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:18.708934 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:18.709305 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:18.709325 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.710577 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.710584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:18.711958 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:18.711969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.712626 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.712630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:18.712757 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:18.713374 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:18.713394 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:18.715199 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:18.717173 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:18.717182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:18.717192 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:18.717501 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:18.717882 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:18.717884 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:18.717885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:18.717985 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:18.718725 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:18.718730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:18.718870 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 10:52:18.719034 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 10:52:18.719420 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":116,"thread_start_us":118,"threads_started":1}
I20260927 10:52:18.722853 19668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:52:18.723680 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.723376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:37921 (local address 127.0.0.1:48290)
0927 10:52:18.723509 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.723592 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 10:52:18.723918 19616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
I20260927 10:52:18.724866 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.724592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:37921 (local address 127.0.0.1:48304)
0927 10:52:18.724727 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.724783 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 10:52:18.725081 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.189:37921: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab.
Entry for principal kudu/127.6.31.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab.
I20260927 10:52:18.734241 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.189:37921
--webserver_interface=127.6.31.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab
--principal=kudu/127.6.31.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:39341,127.6.31.189:37921,127.6.31.188:41223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:52:18.774349 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } attempt: 1
I20260927 10:52:18.775317 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.774954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:37921 (local address 127.0.0.1:48308)
0927 10:52:18.775169 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.775234 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 10:52:18.776070 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.775852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:37921 (local address 127.0.0.1:48312)
0927 10:52:18.775963 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.776015 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:52:18.776192 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.189:37921: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
I20260927 10:52:18.844362 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } attempt: 2
W20260927 10:52:18.844247 19675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:18.844523 19675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:18.844590 19675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:52:18.845289 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.844902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:37921 (local address 127.0.0.1:48316)
0927 10:52:18.845111 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.845212 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 10:52:18.846051 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.845826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:37921 (local address 127.0.0.1:48326)
0927 10:52:18.845946 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.845996 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 10:52:18.846212 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.189:37921: Network error: Client connection negotiation failed: client connection to 127.6.31.189:37921: connect: Connection refused (error 111)
W20260927 10:52:18.849184 19675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:18.849263 19675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:18.849305 19675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:18.849336 19675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:18.849368 19675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:18.854704 19675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:39341,127.6.31.189:37921,127.6.31.188:41223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab
--principal=kudu/127.6.31.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.189:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.31.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:18.855837 19675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:18.856880 19675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:18.865314 19681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:18.865401 19680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:18.865372 19683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:18.865700 19675 server_base.cc:1034] running on GCE node
I20260927 10:52:18.866320 19675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:18.867506 19675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:18.868731 19675 hybrid_clock.cc:648] HybridClock initialized: now 1790506338868709 us; error 46 us; skew 500 ppm
Sep 27 10:52:18 dist-test-slave-c60x krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:18.871922 19675 init.cc:377] Logged in from keytab as kudu/127.6.31.189@KRBTEST.COM (short username kudu)
I20260927 10:52:18.872898 19675 webserver.cc:460] Webserver started at http://127.6.31.189:39781/ using document root <none> and password file <none>
I20260927 10:52:18.873464 19675 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:18.873556 19675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:18.873801 19675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:18.875991 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/data/instance:
uuid: "74c7df40fcd94e958fd7e63309201b68"
format_stamp: "Formatted at 2026-09-27 10:52:18 on dist-test-slave-c60x"
server_key: "16575a9043cc23f689fa419447ff81ab"
server_key_iv: "bb97b18a14ad4d46acf102cb07790768"
server_key_version: "encryptionkey@0"
I20260927 10:52:18.876597 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/wal/instance:
uuid: "74c7df40fcd94e958fd7e63309201b68"
format_stamp: "Formatted at 2026-09-27 10:52:18 on dist-test-slave-c60x"
server_key: "16575a9043cc23f689fa419447ff81ab"
server_key_iv: "bb97b18a14ad4d46acf102cb07790768"
server_key_version: "encryptionkey@0"
I20260927 10:52:18.880534 19675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:52:18.883033 19690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:18.884274 19675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:52:18.884433 19675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/wal
uuid: "74c7df40fcd94e958fd7e63309201b68"
format_stamp: "Formatted at 2026-09-27 10:52:18 on dist-test-slave-c60x"
server_key: "16575a9043cc23f689fa419447ff81ab"
server_key_iv: "bb97b18a14ad4d46acf102cb07790768"
server_key_version: "encryptionkey@0"
I20260927 10:52:18.884554 19675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:18.904513 19675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:18.907444 19675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:18.907649 19675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:18.914723 19675 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.189:37921
I20260927 10:52:18.914747 19742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.189:37921 every 8 connection(s)
I20260927 10:52:18.915732 19675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
I20260927 10:52:18.918474 19743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:18.920524 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 19675
I20260927 10:52:18.920620 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-1/wal/instance
I20260927 10:52:18.920874 6270 external_mini_cluster.cc:1428] Setting key 3c7d70ba69e609dca3d06bbe6dd5ab81
I20260927 10:52:18.922535 19743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.31.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.188@KRBTEST.COM" created.
Sep 27 10:52:18 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:18.937390 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.924874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.6.31.190:39341)
0927 10:52:18.925013 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:18.925017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:18.926028 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 10:52:18.927040 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:18.927044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:18.927133 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:18.927251 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:18.928371 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.928971 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.930185 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.930438 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.930560 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:18.933114 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:18.933118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:18.933120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:18.933144 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:18.934949 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:18.935525 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:18.935530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:18.935532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:18.935631 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:18.936061 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:18.936066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:18.936071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:18.936342 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:18.936464 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:18.937029 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 10:52:18.937163 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:52:18.937398 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } attempt: 3
I20260927 10:52:18.938221 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.925296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:39341 (local address 127.0.0.1:39492)
0927 10:52:18.925879 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.925919 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:18.926821 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:18.927414 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:18.927421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:18.927755 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:18.928226 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:18.928239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.929099 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.929108 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:18.930051 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:18.930060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.930679 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.930684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:18.930822 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:18.931508 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:18.931532 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:18.932939 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:18.935115 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:18.935123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:18.935139 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:18.935416 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:18.935792 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:18.935795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:18.935796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:18.935936 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:18.936460 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:18.936465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:18.936762 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 10:52:18.936926 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 10:52:18.937180 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":166,"threads_started":1}
I20260927 10:52:18.940099 19743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:52:18 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
Sep 27 10:52:18 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
I20260927 10:52:18.948865 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.937998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:37921 (local address 127.0.0.1:48336)
0927 10:52:18.938170 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.938184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:18.938272 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:18.938918 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:18.938921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:18.939148 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:18.939707 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:18.939716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.941015 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.941018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:18.941734 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:18.941744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.942338 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.942341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:18.942456 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:18.942980 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:18.942991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:18.944476 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:18.946951 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:18.946954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:18.946955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:18.947244 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:18.947621 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:18.947624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:18.947625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:18.947703 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:18.948300 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:18.948303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:18.948411 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 10:52:18.948517 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:52:18.948668 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab.
Entry for principal kudu/127.6.31.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab.
I20260927 10:52:18.949023 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.938151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.6.31.189:37921)
0927 10:52:18.938520 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:18.938531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:18.938552 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:52:18.938589 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:18.938594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:18.938789 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:18.938936 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:18.939822 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.940885 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.941846 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.942218 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.942337 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:18.944611 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:18.944620 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:18.944640 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:18.944661 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:18.946807 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:18.947417 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:18.947425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:18.947427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:18.947524 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:18.947813 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:18.947817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:18.947819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:18.948192 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:18.948339 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:18.948627 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 10:52:18.948782 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":145,"threads_started":1}
I20260927 10:52:18.949657 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.940594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:37921 (local address 127.0.0.1:48348)
0927 10:52:18.940703 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.940718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:18.940789 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:18.941136 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:18.941138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:18.941322 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:18.941398 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:18.941403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.942331 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.942334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:18.943044 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:18.943053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.943531 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.943534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:18.943618 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:18.944081 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:18.944094 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:18.945529 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:18.948125 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:18.948128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:18.948129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:18.948316 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:18.948637 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:18.948642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:18.948643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:18.948720 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:18.949185 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:18.949191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:18.949297 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:52:18.949406 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:52:18.949551 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:52:18.950136 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.188:41223
--webserver_interface=127.6.31.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab
--principal=kudu/127.6.31.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:39341,127.6.31.189:37921,127.6.31.188:41223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:52:18.950613 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.940645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.6.31.189:37921)
0927 10:52:18.940956 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:18.940962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:18.940971 (+ 9us) server_negotiation.cc:407] Connection header received
0927 10:52:18.940995 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:18.940997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:18.941026 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:18.941128 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:18.941504 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.942214 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.943196 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:18.943424 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:18.943531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:18.945722 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:18.945726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:18.945729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:18.945756 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:18.948032 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:18.948435 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:18.948439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:18.948441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:18.948522 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:18.948837 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:18.948843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:18.948845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:18.949041 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:18.949149 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:18.950237 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0927 10:52:18.950361 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260927 10:52:18.951128 19668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:52:18.951722 19743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:52:18.951946 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.951629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39290)
0927 10:52:18.951793 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.951872 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 10:52:18.952418 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.952144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39300)
0927 10:52:18.952265 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.952343 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:52:18.952606 19694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
I20260927 10:52:18.952703 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.952485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39304)
0927 10:52:18.952601 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.952646 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 10:52:18.952844 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:41223: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
I20260927 10:52:18.953469 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.953272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39318)
0927 10:52:18.953373 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.953414 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:52:18.953619 19743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:41223: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
I20260927 10:52:18.984016 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } attempt: 1
I20260927 10:52:18.984890 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.984588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39324)
0927 10:52:18.984748 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.984813 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 10:52:18.985664 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:18.985443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39326)
0927 10:52:18.985560 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:18.985607 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:52:18.985827 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:41223: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
I20260927 10:52:19.002885 19743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } attempt: 1
I20260927 10:52:19.003801 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.003506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39342)
0927 10:52:19.003663 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.003725 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 10:52:19.004526 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.004326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39354)
0927 10:52:19.004430 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.004479 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:52:19.004652 19743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:41223: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
I20260927 10:52:19.061005 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } attempt: 2
I20260927 10:52:19.061858 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.061531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39356)
0927 10:52:19.061712 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.061784 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 10:52:19.062573 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.062357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39360)
0927 10:52:19.062464 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.062515 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:52:19.062728 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:41223: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
W20260927 10:52:19.063877 19750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:19.064123 19750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:19.064208 19750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:19.068306 19750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:19.068360 19750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:19.068380 19750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:19.068397 19750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:19.068436 19750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:19.072824 19743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } attempt: 2
I20260927 10:52:19.073747 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.073377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39376)
0927 10:52:19.073525 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.073598 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 10:52:19.074076 19750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:39341,127.6.31.189:37921,127.6.31.188:41223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab
--principal=kudu/127.6.31.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.188:41223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.31.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:19.074579 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.074342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39378)
0927 10:52:19.074470 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.074520 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 10:52:19.074780 19743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:41223: Network error: Client connection negotiation failed: client connection to 127.6.31.188:41223: connect: Connection refused (error 111)
I20260927 10:52:19.075409 19750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:19.076447 19750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:52:19.085187 19750 server_base.cc:1034] running on GCE node
W20260927 10:52:19.085148 19757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:19.085170 19759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:19.085141 19756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:19.085994 19750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:19.087134 19750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:19.088362 19750 hybrid_clock.cc:648] HybridClock initialized: now 1790506339088343 us; error 31 us; skew 500 ppm
Sep 27 10:52:19 dist-test-slave-c60x krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506339, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:19.091269 19750 init.cc:377] Logged in from keytab as kudu/127.6.31.188@KRBTEST.COM (short username kudu)
I20260927 10:52:19.092193 19750 webserver.cc:460] Webserver started at http://127.6.31.188:40303/ using document root <none> and password file <none>
I20260927 10:52:19.092777 19750 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:19.092867 19750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:19.093080 19750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:19.095191 19750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/data/instance:
uuid: "cc67b00eb0be4bb782d66ff0b6ffa008"
format_stamp: "Formatted at 2026-09-27 10:52:19 on dist-test-slave-c60x"
server_key: "fe5811c6d0ca963216a80330a14bf76c"
server_key_iv: "e2f52e2791e5f0239026ae3aa5582702"
server_key_version: "encryptionkey@0"
I20260927 10:52:19.095777 19750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/wal/instance:
uuid: "cc67b00eb0be4bb782d66ff0b6ffa008"
format_stamp: "Formatted at 2026-09-27 10:52:19 on dist-test-slave-c60x"
server_key: "fe5811c6d0ca963216a80330a14bf76c"
server_key_iv: "e2f52e2791e5f0239026ae3aa5582702"
server_key_version: "encryptionkey@0"
I20260927 10:52:19.099535 19750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:52:19.101854 19766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:19.103313 19750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:52:19.103444 19750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/wal
uuid: "cc67b00eb0be4bb782d66ff0b6ffa008"
format_stamp: "Formatted at 2026-09-27 10:52:19 on dist-test-slave-c60x"
server_key: "fe5811c6d0ca963216a80330a14bf76c"
server_key_iv: "e2f52e2791e5f0239026ae3aa5582702"
server_key_version: "encryptionkey@0"
I20260927 10:52:19.103555 19750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:19.123191 19750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:19.126206 19750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:19.126426 19750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:19.133406 19750 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.188:41223
I20260927 10:52:19.133423 19818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.188:41223 every 8 connection(s)
I20260927 10:52:19.134483 19750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
I20260927 10:52:19.137225 19819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:19.137734 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 19750
I20260927 10:52:19.137868 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506175715859-6270-0/minicluster-data/master-2/wal/instance
I20260927 10:52:19.138136 6270 external_mini_cluster.cc:1428] Setting key d4723becfae0bc183c82291a8b61dd46
I20260927 10:52:19.141167 19819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:52:19 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
Sep 27 10:52:19 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506339, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:19.152211 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.143045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.6.31.190:39341)
0927 10:52:19.143226 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.143231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.143480 (+ 249us) server_negotiation.cc:407] Connection header received
0927 10:52:19.143614 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.143616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.143659 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.143722 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.144521 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.144993 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.146082 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.146305 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.146414 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.148802 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.148806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.148807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.148829 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.150463 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.150986 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.150989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.150991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.151076 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.151420 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.151423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.151424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.151600 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.151676 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.152017 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.152104 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 10:52:19.152241 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.139409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.6.31.188:41223)
0927 10:52:19.139791 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.139805 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.139838 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:52:19.140451 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.140466 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.140760 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.141015 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.141986 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.143200 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.144353 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.144801 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.144971 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.147416 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.147423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.147434 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.147458 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.149317 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.149821 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.149828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.149835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.149957 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.150235 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.150239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.150240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.150573 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.150773 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.151035 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.151335 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":126,"threads_started":1}
I20260927 10:52:19.152276 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.142958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:39341 (local address 127.0.0.1:39494)
0927 10:52:19.143337 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.143388 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:19.143510 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:19.143804 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:19.143809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:19.144214 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:19.144399 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.144407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.145132 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.145136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.145950 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.145960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.146449 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.146453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.146560 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.147064 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:19.147118 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:19.148652 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:19.150565 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.150575 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.150580 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.150881 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.151208 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.151211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.151213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.151324 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.151706 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:19.151711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:19.151817 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:52:19.151955 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 10:52:19.152118 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
I20260927 10:52:19.154079 19819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:52:19 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506339, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
Sep 27 10:52:19 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:19.161890 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } attempt: 3
I20260927 10:52:19.163036 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.154563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:37921 (local address 127.0.0.1:48360)
0927 10:52:19.154875 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.154890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:19.154951 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:19.155199 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:19.155201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:19.155368 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:19.155769 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.155775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.156491 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.156497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.156998 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.157005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.157530 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.157535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.157648 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.158013 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:19.158033 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:19.159284 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:19.161250 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.161254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.161256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.161516 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.161831 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.161834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.161836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.161930 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.162474 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:19.162477 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:19.162566 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 10:52:19.162701 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 10:52:19.162851 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260927 10:52:19.163126 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.154812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.6.31.189:37921)
0927 10:52:19.154954 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.154958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.154974 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:52:19.155020 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.155023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.155068 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.155188 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.155886 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.156386 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.157113 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.157401 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.157510 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.159416 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.159419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.159422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.159450 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.161142 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.161617 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.161619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.161620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.161693 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.162081 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.162085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.162088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.162332 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.162429 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.162798 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.162948 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:52:19.164253 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.154652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.6.31.190:39341)
0927 10:52:19.154743 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.154748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.154764 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:52:19.154801 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.154803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.154847 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.154925 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.155549 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.156016 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.156793 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.157004 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.157108 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.160067 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.160070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.160072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.160093 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.161970 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.162472 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.162475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.162477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.162561 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.162884 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.162887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.162889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.163121 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.163344 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.163655 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.164096 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 10:52:19.164215 19819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:52:19.165956 19743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } attempt: 3
Sep 27 10:52:19 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
Sep 27 10:52:19 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
Sep 27 10:52:19 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
I20260927 10:52:19.172456 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.162318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39398)
0927 10:52:19.162497 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.162512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:19.162576 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:19.162861 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:19.162863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:19.163033 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:19.163179 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.163188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.164261 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.164264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.165016 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.165024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.166146 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.166149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.166252 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.166773 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:19.166784 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:19.168130 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:19.170229 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.170232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.170235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.170506 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.170987 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.170990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.170992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.171111 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.171904 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:19.171907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:19.172013 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:52:19.172106 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:52:19.172231 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
Sep 27 10:52:19 dist-test-slave-c60x krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506339, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
I20260927 10:52:19.172911 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.162416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.6.31.188:41223)
0927 10:52:19.162522 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.162526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.162541 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:52:19.162672 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.162678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.162725 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.163303 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.163378 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.164135 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.165456 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.165777 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.168316 (+ 2539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.168413 (+ 97us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.168417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.168419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.168435 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.170094 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.170792 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.170798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.170803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.170890 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.171230 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.171233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.171235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.171454 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.172637 (+ 1183us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.172711 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.172810 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
I20260927 10:52:19.177273 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.166299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39422)
0927 10:52:19.166457 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.166471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:19.166543 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:19.167210 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:19.167216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:19.167400 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:19.167495 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.167506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.168599 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.168602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.169587 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.169591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.170241 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.170244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.170339 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.170794 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:19.170810 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:19.173422 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:19.175640 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.175644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.175646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.175884 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.176190 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.176194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.176197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.176288 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.176726 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:19.176729 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:19.176844 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 10:52:19.176954 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 10:52:19.177105 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":240}
I20260927 10:52:19.177287 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.166505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.6.31.188:41223)
0927 10:52:19.166932 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.166947 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.166962 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:52:19.166997 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.166999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.167037 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.167189 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.167631 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.168476 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.169696 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.170078 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.170551 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.173565 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.173568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.173571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.173594 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.175520 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.175997 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.176000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.176002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.176079 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.176379 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.176381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.176383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.176637 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.176724 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.177004 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.177115 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"spinlock_wait_cycles":768,"thread_start_us":149,"threads_started":1}
I20260927 10:52:19.177563 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.165286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:41223 (local address 127.0.0.1:39414)
0927 10:52:19.165930 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:19.165943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:19.166014 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:19.167921 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:19.167924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:19.168088 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:19.168176 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.168185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.169770 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.169773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.170757 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:19.170771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.171389 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.171392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:19.171460 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.171911 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:19.171924 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:19.173451 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:19.175884 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.175888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.175890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.176119 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.176434 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:19.176439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:19.176444 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:19.176512 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:19.177025 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:19.177027 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:19.177118 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:52:19.177258 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 10:52:19.177419 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581}
I20260927 10:52:19.177973 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.165456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.6.31.188:41223)
0927 10:52:19.165821 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.165830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.165987 (+ 157us) server_negotiation.cc:407] Connection header received
0927 10:52:19.166129 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.166135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.166183 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.166274 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.168271 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.169654 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.170874 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.171257 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.171344 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.173581 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.173585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.173587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.173624 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.175777 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.176220 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.176223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.176225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.176335 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.176623 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.176626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.176628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.176896 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.176989 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.177723 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.177821 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":268,"spinlock_wait_cycles":768,"thread_start_us":135,"threads_started":1}
I20260927 10:52:19.177955 19668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c: Bootstrap starting.
I20260927 10:52:19.180660 19819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008: Bootstrap starting.
I20260927 10:52:19.180794 19668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:19.180794 19743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68: Bootstrap starting.
I20260927 10:52:19.181677 19668 log.cc:824] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:19.182853 19743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:19.183020 19819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:19.183638 19743 log.cc:824] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:19.183835 19819 log.cc:824] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:19.183992 19668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c: No bootstrap required, opened a new log
I20260927 10:52:19.185612 19743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68: No bootstrap required, opened a new log
I20260927 10:52:19.186971 19819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008: No bootstrap required, opened a new log
I20260927 10:52:19.187378 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.165354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.6.31.189:37921)
0927 10:52:19.165455 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.165459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.165820 (+ 361us) server_negotiation.cc:407] Connection header received
0927 10:52:19.165919 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.165922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.165972 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.166047 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.167334 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.168010 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.169081 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.169414 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.169470 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.174079 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.174082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.174084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.174108 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.185204 (+ 11096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.185845 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.185849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.185852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.185939 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.186269 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.186272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.186274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.186506 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.186655 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.187053 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.187218 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":29}
I20260927 10:52:19.188027 19668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } }
I20260927 10:52:19.188277 19668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:19.188345 19668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72feaf6673f448daa28c6a3d1d3fd44c, State: Initialized, Role: FOLLOWER
I20260927 10:52:19.189147 19668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } }
I20260927 10:52:19.189767 19829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.189991 19829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:52:19.190059 19743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } }
I20260927 10:52:19.190297 19743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:19.190361 19743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c7df40fcd94e958fd7e63309201b68, State: Initialized, Role: FOLLOWER
I20260927 10:52:19.190596 19668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:52:19.190807 19743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } }
I20260927 10:52:19.191468 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:52:19.192513 19743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:52:19.196133 19842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:52:19.196223 19842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 10:52:19.197921 19859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:52:19.198055 19859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:52:19.198287 19831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.198413 19831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:52:19.200197 19819 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } }
I20260927 10:52:19.200454 19819 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:19.200508 19819 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc67b00eb0be4bb782d66ff0b6ffa008, State: Initialized, Role: FOLLOWER
I20260927 10:52:19.200989 19819 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } }
I20260927 10:52:19.202311 19819 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:52:19.199527 19600 master_runner.cc:186] Error getting master registration for 127.6.31.188:41223: OK, instance_id { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" instance_seqno: 1790506339131447 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:52:19.202796 19600 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:52:19.199759 19675 master_runner.cc:186] Error getting master registration for 127.6.31.188:41223: OK, instance_id { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" instance_seqno: 1790506339131447 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:52:19.203253 19675 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:52:19.203488 19862 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.203636 19862 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 10:52:19.207245 19873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:52:19.207324 19873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:52:19.209682 19750 master_runner.cc:263] No leader master found from master cc67b00eb0be4bb782d66ff0b6ffa008
I20260927 10:52:19.209802 19750 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:52:19.219105 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.197413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.6.31.188:41223)
0927 10:52:19.197526 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.197530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.210067 (+ 12537us) server_negotiation.cc:407] Connection header received
0927 10:52:19.210176 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.210181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.210229 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.210296 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.211176 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.211939 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.213051 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.213417 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.213497 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.215403 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.215407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.215410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.215435 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.217202 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.217731 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.217747 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.217751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.217837 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.218130 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.218133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.218135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.218340 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.218455 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.218804 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.218923 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:52:19.227929 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.193653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.6.31.190:39341)
0927 10:52:19.193785 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.193790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.207143 (+ 13353us) server_negotiation.cc:407] Connection header received
0927 10:52:19.207369 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.207373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.207421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.207473 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.210219 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.211029 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.219717 (+ 8688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.219993 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.220054 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.224027 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.224031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.224033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.224064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.225688 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.226191 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.226194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.226196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.226293 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.226630 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.226634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.226636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.226869 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.226991 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.227618 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.227734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:52:19.229982 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:19.194248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.6.31.189:37921)
0927 10:52:19.194345 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:19.194350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:19.209844 (+ 15494us) server_negotiation.cc:407] Connection header received
0927 10:52:19.219331 (+ 9487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:19.219336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:19.219391 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:19.219440 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:19.220994 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.221569 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.222294 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:19.222905 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:19.223017 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:19.225525 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:19.225529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:19.225535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:19.225568 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:19.227352 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.227937 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.227944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.227948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.228446 (+ 498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:19.228832 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:19.228836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:19.228837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:19.229038 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:19.229161 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:19.229714 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 10:52:19.229838 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 10:52:19.332337 19831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:19.332536 19831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } }
I20260927 10:52:19.333523 19831 leader_election.cc:288] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72feaf6673f448daa28c6a3d1d3fd44c (127.6.31.190:39341), cc67b00eb0be4bb782d66ff0b6ffa008 (127.6.31.188:41223)
I20260927 10:52:19.334316 19793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74c7df40fcd94e958fd7e63309201b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" is_pre_election: true
I20260927 10:52:19.334653 19793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c7df40fcd94e958fd7e63309201b68 in term 0.
I20260927 10:52:19.335109 19694 leader_election.cc:302] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c7df40fcd94e958fd7e63309201b68, cc67b00eb0be4bb782d66ff0b6ffa008; no voters:
I20260927 10:52:19.335436 19831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:52:19.335532 19831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:52:19.335585 19831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:19.336920 19831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } }
I20260927 10:52:19.337308 19831 leader_election.cc:288] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [CANDIDATE]: Term 1 election: Requested vote from peers 72feaf6673f448daa28c6a3d1d3fd44c (127.6.31.190:39341), cc67b00eb0be4bb782d66ff0b6ffa008 (127.6.31.188:41223)
I20260927 10:52:19.337735 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74c7df40fcd94e958fd7e63309201b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008"
I20260927 10:52:19.337869 19792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:19.339171 19792 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c7df40fcd94e958fd7e63309201b68 in term 1.
I20260927 10:52:19.339498 19694 leader_election.cc:302] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c7df40fcd94e958fd7e63309201b68, cc67b00eb0be4bb782d66ff0b6ffa008; no voters:
I20260927 10:52:19.340385 19642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74c7df40fcd94e958fd7e63309201b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" is_pre_election: true
I20260927 10:52:19.340703 19642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c7df40fcd94e958fd7e63309201b68 in term 0.
I20260927 10:52:19.341017 19641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74c7df40fcd94e958fd7e63309201b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72feaf6673f448daa28c6a3d1d3fd44c"
I20260927 10:52:19.341135 19641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:19.342428 19641 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c7df40fcd94e958fd7e63309201b68 in term 1.
I20260927 10:52:19.343607 19831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:52:19.343874 19831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [term 1 LEADER]: Becoming Leader. State: Replica: 74c7df40fcd94e958fd7e63309201b68, State: Running, Role: LEADER
I20260927 10:52:19.344213 19831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } }
I20260927 10:52:19.345973 19831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c7df40fcd94e958fd7e63309201b68. Latest consensus state: current_term: 1 leader_uuid: "74c7df40fcd94e958fd7e63309201b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.346092 19831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:19.346822 19877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:52:19.347878 19877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:52:19.355604 19792 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [term 1 FOLLOWER]: Refusing update from remote peer 74c7df40fcd94e958fd7e63309201b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:19.356294 19831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:19.360481 19862 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c7df40fcd94e958fd7e63309201b68. Latest consensus state: current_term: 1 leader_uuid: "74c7df40fcd94e958fd7e63309201b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.360631 19862 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:52:19.364006 19641 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [term 1 FOLLOWER]: Refusing update from remote peer 74c7df40fcd94e958fd7e63309201b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:19.364588 19831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:19.368634 19829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c7df40fcd94e958fd7e63309201b68. Latest consensus state: current_term: 1 leader_uuid: "74c7df40fcd94e958fd7e63309201b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.368775 19829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:52:19.372386 19875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74c7df40fcd94e958fd7e63309201b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.372539 19875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:19.373700 19862 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74c7df40fcd94e958fd7e63309201b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.373828 19862 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:52:19.374470 19878 mvcc.cc:205] Tried to move back new op lower bound from 7333913965992640512 to 7333913965956976640. Current Snapshot: MvccSnapshot[applied={T|T < 7333913965992640512}]
I20260927 10:52:19.376066 19831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74c7df40fcd94e958fd7e63309201b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.376209 19831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:19.379549 19829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74c7df40fcd94e958fd7e63309201b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72feaf6673f448daa28c6a3d1d3fd44c" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39341 } } peers { permanent_uuid: "74c7df40fcd94e958fd7e63309201b68" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 37921 } } peers { permanent_uuid: "cc67b00eb0be4bb782d66ff0b6ffa008" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 41223 } } }
I20260927 10:52:19.379670 19829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:52:19.380693 19879 mvcc.cc:205] Tried to move back new op lower bound from 7333913965992640512 to 7333913965956976640. Current Snapshot: MvccSnapshot[applied={T|T < 7333913965992640512}]
I20260927 10:52:19.383890 19877 catalog_manager.cc:1347] Generated new cluster ID: 7ec27602ad8a40c59169506d8a1c227d
I20260927 10:52:19.383963 19877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:52:19.397706 19877 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:52:19.399060 19877 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:52:19.407905 19877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74c7df40fcd94e958fd7e63309201b68: Generated new TSK 0
I20260927 10:52:19.408776 19877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:52:20.197594 19842 catalog_manager.cc:1259] Loaded cluster ID: 7ec27602ad8a40c59169506d8a1c227d
I20260927 10:52:20.197697 19842 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c: loading cluster ID for follower catalog manager: success
I20260927 10:52:20.199487 19842 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c: acquiring CA information for follower catalog manager: success
I20260927 10:52:20.200635 19842 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72feaf6673f448daa28c6a3d1d3fd44c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:52:20.208879 19873 catalog_manager.cc:1259] Loaded cluster ID: 7ec27602ad8a40c59169506d8a1c227d
I20260927 10:52:20.208958 19873 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008: loading cluster ID for follower catalog manager: success
I20260927 10:52:20.210664 19873 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008: acquiring CA information for follower catalog manager: success
I20260927 10:52:20.211876 19873 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cc67b00eb0be4bb782d66ff0b6ffa008: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:52:20.470178 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 19600
I20260927 10:52:20.476719 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 19675
I20260927 10:52:20.482774 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 19750
2026-09-27T10:52:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3142 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:52:20 dist-test-slave-c60x krb5kdc[19892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:52:20 dist-test-slave-c60x krb5kdc[19892](info): set up 2 sockets
Sep 27 10:52:20 dist-test-slave-c60x krb5kdc[19892](info): commencing operation
krb5kdc: starting...
W20260927 10:52:21.543515 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:52:21 dist-test-slave-c60x krb5kdc[19892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:52:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:52:21.670603 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35317
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:35317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:21.780110 19905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:21.780427 19905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:21.780503 19905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:21.784587 19905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:21.784688 19905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:21.784727 19905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:21.784765 19905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:21.784799 19905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:21.790198 19905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:35317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:21.791366 19905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:21.792397 19905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:21.800937 19913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:21.800915 19911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:21.801144 19905 server_base.cc:1034] running on GCE node
W20260927 10:52:21.800999 19910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:21.801838 19905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:21.802937 19905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:21.804104 19905 hybrid_clock.cc:648] HybridClock initialized: now 1790506341804084 us; error 46 us; skew 500 ppm
Sep 27 10:52:21 dist-test-slave-c60x krb5kdc[19892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506341, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:21.806980 19905 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:52:21.807902 19905 webserver.cc:460] Webserver started at http://127.6.31.190:45243/ using document root <none> and password file <none>
I20260927 10:52:21.808460 19905 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:21.808521 19905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:21.808763 19905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:21.810859 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "5fce43cb36124fd29f5b3b02eba97c56"
format_stamp: "Formatted at 2026-09-27 10:52:21 on dist-test-slave-c60x"
server_key: "6ac931ab7334e139875c065dd32f87f0"
server_key_iv: "a99a83589b54eec25711be87f6f3b27e"
server_key_version: "encryptionkey@0"
I20260927 10:52:21.811473 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "5fce43cb36124fd29f5b3b02eba97c56"
format_stamp: "Formatted at 2026-09-27 10:52:21 on dist-test-slave-c60x"
server_key: "6ac931ab7334e139875c065dd32f87f0"
server_key_iv: "a99a83589b54eec25711be87f6f3b27e"
server_key_version: "encryptionkey@0"
I20260927 10:52:21.815264 19905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:52:21.817812 19920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:21.819190 19905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:52:21.819320 19905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "5fce43cb36124fd29f5b3b02eba97c56"
format_stamp: "Formatted at 2026-09-27 10:52:21 on dist-test-slave-c60x"
server_key: "6ac931ab7334e139875c065dd32f87f0"
server_key_iv: "a99a83589b54eec25711be87f6f3b27e"
server_key_version: "encryptionkey@0"
I20260927 10:52:21.819437 19905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:21.845067 19905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:21.848537 19905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:21.848760 19905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:21.856382 19905 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:35317
I20260927 10:52:21.856400 19972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:35317 every 8 connection(s)
I20260927 10:52:21.857430 19905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:52:21.860214 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:21.865756 19973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56: Bootstrap starting.
I20260927 10:52:21.867221 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 19905
I20260927 10:52:21.867316 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:52:21.867583 6270 external_mini_cluster.cc:1428] Setting key 40e31b81591ecb13ad762c77f905adda
I20260927 10:52:21.868386 19973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:21.869279 19973 log.cc:824] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:21.871389 19973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56: No bootstrap required, opened a new log
I20260927 10:52:21.875715 19973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce43cb36124fd29f5b3b02eba97c56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35317 } }
I20260927 10:52:21.875928 19973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:21.875962 19973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fce43cb36124fd29f5b3b02eba97c56, State: Initialized, Role: FOLLOWER
I20260927 10:52:21.876457 19973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce43cb36124fd29f5b3b02eba97c56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35317 } }
I20260927 10:52:21.876636 19973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:52:21.876721 19973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 27 10:52:21 dist-test-slave-c60x krb5kdc[19892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:21.876843 19973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:21.877970 19973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce43cb36124fd29f5b3b02eba97c56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35317 } }
I20260927 10:52:21.878363 19973 leader_election.cc:302] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fce43cb36124fd29f5b3b02eba97c56; no voters:
I20260927 10:52:21.878765 19973 leader_election.cc:288] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:52:21.878893 19978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:52:21.879158 19978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [term 1 LEADER]: Becoming Leader. State: Replica: 5fce43cb36124fd29f5b3b02eba97c56, State: Running, Role: LEADER
I20260927 10:52:21.879531 19978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce43cb36124fd29f5b3b02eba97c56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35317 } }
I20260927 10:52:21.880105 19973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:52:21.880879 19978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fce43cb36124fd29f5b3b02eba97c56. Latest consensus state: current_term: 1 leader_uuid: "5fce43cb36124fd29f5b3b02eba97c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce43cb36124fd29f5b3b02eba97c56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35317 } } }
I20260927 10:52:21.881088 19978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:21.881323 19979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fce43cb36124fd29f5b3b02eba97c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce43cb36124fd29f5b3b02eba97c56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35317 } } }
I20260927 10:52:21.881451 19979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:21.881596 19987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:52:21.883900 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:21.868628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.6.31.190:35317)
0927 10:52:21.869098 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:21.869118 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:21.869174 (+ 56us) server_negotiation.cc:407] Connection header received
0927 10:52:21.869940 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:21.869987 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:21.870377 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:21.870721 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:21.871460 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:21.872944 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:21.873773 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:21.874583 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:21.874743 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:21.877456 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:21.877462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:21.877478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:21.877509 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:21.879657 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:21.880197 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:21.880203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:21.880209 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:21.880344 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:21.880660 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:21.880663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:21.880666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:21.881372 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:21.882402 (+ 1030us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:21.882613 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 10:52:21.882911 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":146,"threads_started":1}
I20260927 10:52:21.884891 19987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:52:21.887265 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:52:21.891844 19987 catalog_manager.cc:1347] Generated new cluster ID: d03a813f3df54c00a0a4dd49cc63b3c8
I20260927 10:52:21.891920 19987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:52:21.897838 19987 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:52:21.898679 19987 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:52:21.907903 19987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fce43cb36124fd29f5b3b02eba97c56: Generated new TSK 0
I20260927 10:52:21.908583 19987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:52:21.943542 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 19905
2026-09-27T10:52:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1458 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:52:22 dist-test-slave-c60x krb5kdc[20000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:52:22 dist-test-slave-c60x krb5kdc[20000](info): set up 2 sockets
Sep 27 10:52:22 dist-test-slave-c60x krb5kdc[20000](info): commencing operation
krb5kdc: starting...
W20260927 10:52:22.999593 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:52:23 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:52:23.126631 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36581
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:36581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:23.238999 20013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:23.239382 20013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:23.239425 20013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:23.243587 20013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:23.243659 20013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:23.243678 20013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:23.243695 20013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:23.243747 20013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:23.249495 20013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:36581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:23.250638 20013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:23.251761 20013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:23.260702 20021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:23.260866 20013 server_base.cc:1034] running on GCE node
W20260927 10:52:23.260776 20019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:23.261037 20018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:23.261708 20013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:23.262909 20013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:23.264096 20013 hybrid_clock.cc:648] HybridClock initialized: now 1790506343264084 us; error 28 us; skew 500 ppm
Sep 27 10:52:23 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506343, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:23.276407 20013 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:52:23.277420 20013 webserver.cc:460] Webserver started at http://127.6.31.190:44665/ using document root <none> and password file <none>
I20260927 10:52:23.277963 20013 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:23.278023 20013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:23.278194 20013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:23.280349 20013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "8976e07ff09c4e16b58dbc2b512a4504"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "333f1f36aed65d93211b3493c8ee588e"
server_key_iv: "05f066dae6ef62a9b10e9404593f9df9"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.280864 20013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "8976e07ff09c4e16b58dbc2b512a4504"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "333f1f36aed65d93211b3493c8ee588e"
server_key_iv: "05f066dae6ef62a9b10e9404593f9df9"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.284737 20013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:52:23.287266 20028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:23.288538 20013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:52:23.288652 20013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "8976e07ff09c4e16b58dbc2b512a4504"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "333f1f36aed65d93211b3493c8ee588e"
server_key_iv: "05f066dae6ef62a9b10e9404593f9df9"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.288795 20013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:23.309187 20013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:23.312647 20013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:23.312865 20013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:23.320493 20013 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:36581
I20260927 10:52:23.320811 20080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:36581 every 8 connection(s)
I20260927 10:52:23.321492 20013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:52:23.323609 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 20013
I20260927 10:52:23.323710 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:52:23.324043 6270 external_mini_cluster.cc:1428] Setting key 1915351c84fc77b90b311eb9e2c472a4
I20260927 10:52:23.324564 20081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:23.331748 20081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Bootstrap starting.
Sep 27 10:52:23 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:23.334116 20081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:23.335132 20081 log.cc:824] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:23.337190 20081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: No bootstrap required, opened a new log
I20260927 10:52:23.339504 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:23.325478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.6.31.190:36581)
0927 10:52:23.325912 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:23.325937 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:23.325979 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:52:23.326803 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:23.326825 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:23.327271 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:23.327628 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:23.328337 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.329831 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.330721 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.331418 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.331573 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:23.334058 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:23.334068 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:23.334087 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:23.334122 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:23.336164 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:23.336673 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:23.336679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:23.336685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:23.336801 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:23.337177 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:23.337182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:23.337185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:23.337884 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:23.338081 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:23.338419 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 10:52:23.338718 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":126,"threads_started":1}
I20260927 10:52:23.341140 20081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:23.341388 20081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:23.341480 20081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8976e07ff09c4e16b58dbc2b512a4504, State: Initialized, Role: FOLLOWER
I20260927 10:52:23.341890 20081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:23.342011 20081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:52:23.342077 20081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:52:23.342149 20081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:23.343181 20081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:23.343546 20081 leader_election.cc:302] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8976e07ff09c4e16b58dbc2b512a4504; no voters:
I20260927 10:52:23.343883 20081 leader_election.cc:288] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:52:23.343993 20086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:52:23.344244 20086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 1 LEADER]: Becoming Leader. State: Replica: 8976e07ff09c4e16b58dbc2b512a4504, State: Running, Role: LEADER
I20260927 10:52:23.344599 20086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:23.345091 20081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:52:23.346212 20088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8976e07ff09c4e16b58dbc2b512a4504. Latest consensus state: current_term: 1 leader_uuid: "8976e07ff09c4e16b58dbc2b512a4504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } } }
I20260927 10:52:23.346410 20088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:23.346876 20095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:52:23.346822 20087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8976e07ff09c4e16b58dbc2b512a4504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } } }
I20260927 10:52:23.347002 20087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:23.349601 20095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:52:23.354944 20095 catalog_manager.cc:1347] Generated new cluster ID: 1ede30254f5d42a6987e58824736f3c1
I20260927 10:52:23.355014 20095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:52:23.367329 20095 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:52:23.368628 20095 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:52:23.379012 20095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Generated new TSK 0
I20260927 10:52:23.379967 20095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:52:23.416745 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36581
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:23.528771 20107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:23.529028 20107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:23.529069 20107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:23.533070 20107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:23.533130 20107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:23.533188 20107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:52:23.538362 20107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:23.539374 20107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:23.540323 20107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:23.550144 20113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:23.550140 20112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:23.550295 20107 server_base.cc:1034] running on GCE node
W20260927 10:52:23.550140 20115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:23.550858 20107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:23.551594 20107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:23.552776 20107 hybrid_clock.cc:648] HybridClock initialized: now 1790506343552743 us; error 46 us; skew 500 ppm
Sep 27 10:52:23 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506343, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:23.555608 20107 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:52:23.556492 20107 webserver.cc:460] Webserver started at http://127.6.31.129:38395/ using document root <none> and password file <none>
I20260927 10:52:23.557106 20107 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:23.557199 20107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:23.557421 20107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:23.559505 20107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "42766c759ce44956a5ffcd71e43f36b1"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "34c10f8c1001f0b5860a264ae7b1865d"
server_key_iv: "e9b5546ecec134278e7251c2eb13f335"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.560066 20107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "42766c759ce44956a5ffcd71e43f36b1"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "34c10f8c1001f0b5860a264ae7b1865d"
server_key_iv: "e9b5546ecec134278e7251c2eb13f335"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.563936 20107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:52:23.566354 20122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:23.567806 20107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:52:23.567948 20107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "42766c759ce44956a5ffcd71e43f36b1"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "34c10f8c1001f0b5860a264ae7b1865d"
server_key_iv: "e9b5546ecec134278e7251c2eb13f335"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.568064 20107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:23.584409 20107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:23.587539 20107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:23.587749 20107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:23.588385 20107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:23.589213 20107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:23.589285 20107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:23.589345 20107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:23.589392 20107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:23.598704 20107 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:34075
I20260927 10:52:23.598727 20235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:34075 every 8 connection(s)
I20260927 10:52:23.599647 20107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:52:23.603503 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 20107
I20260927 10:52:23.603627 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:52:23.603912 6270 external_mini_cluster.cc:1428] Setting key 1eeb25a63a2bda9fac200c60cd9bac77
Sep 27 10:52:23 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506343, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:23.613750 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:23.601348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:50165 (local address 127.6.31.190:36581)
0927 10:52:23.601495 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:23.601500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:23.602564 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 10:52:23.603406 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:23.603409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:23.603470 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:23.603573 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:23.604931 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.605467 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.606578 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.606815 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.606926 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:23.609621 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:23.609626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:23.609629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:23.609660 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:23.611461 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:23.611998 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:23.612003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:23.612007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:23.612107 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:23.612463 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:23.612466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:23.612468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:23.612724 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:23.612835 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:23.613330 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 10:52:23.613477 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:52:23.614498 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:23.601756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:50165)
0927 10:52:23.602411 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:23.602454 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:23.603201 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:23.603818 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:23.603824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:23.604230 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:23.604789 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:23.604803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.605592 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.605595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:23.606453 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:23.606462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.606962 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.606966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:23.607192 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:23.607856 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:23.607872 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:23.609436 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:23.611600 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:23.611604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:23.611617 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:23.611892 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:23.612205 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:23.612208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:23.612209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:23.612366 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:23.612833 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:23.612838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:23.613076 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 10:52:23.613234 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 10:52:23.613527 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":115,"threads_started":1}
I20260927 10:52:23.615988 20236 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36581
I20260927 10:52:23.616199 20236 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:23.616722 20236 heartbeater.cc:511] Master 127.6.31.190:36581 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:52:23.618265 20045 ts_manager.cc:194] Registered new tserver with Master: 42766c759ce44956a5ffcd71e43f36b1 (127.6.31.129:34075)
I20260927 10:52:23.619831 20045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:50165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:52:23.634156 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36581
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:23.744059 20241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:23.744408 20241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:23.744488 20241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:23.749187 20241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:23.749297 20241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:23.749399 20241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:52:23.754985 20241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:23.756170 20241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:23.757272 20241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:23.767058 20247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:23.767344 20241 server_base.cc:1034] running on GCE node
W20260927 10:52:23.767068 20246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:23.767058 20249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:23.767908 20241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:23.768602 20241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:23.769783 20241 hybrid_clock.cc:648] HybridClock initialized: now 1790506343769767 us; error 24 us; skew 500 ppm
Sep 27 10:52:23 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506343, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:23.772851 20241 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:52:23.773756 20241 webserver.cc:460] Webserver started at http://127.6.31.130:35975/ using document root <none> and password file <none>
I20260927 10:52:23.774294 20241 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:23.774354 20241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:23.774521 20241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:23.776684 20241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "374a36d475454ef78c4ad803b22f0158"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "2614df765363825ed093a1aa3cec6934"
server_key_iv: "6dad4089fb1fc9d248af6d62a80e0bd1"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.777206 20241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "374a36d475454ef78c4ad803b22f0158"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "2614df765363825ed093a1aa3cec6934"
server_key_iv: "6dad4089fb1fc9d248af6d62a80e0bd1"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.781183 20241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:52:23.783758 20256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:23.785002 20241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:52:23.785140 20241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "374a36d475454ef78c4ad803b22f0158"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "2614df765363825ed093a1aa3cec6934"
server_key_iv: "6dad4089fb1fc9d248af6d62a80e0bd1"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.785279 20241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:23.796823 20241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:23.799634 20241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:23.799844 20241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:23.800501 20241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:23.801388 20241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:23.801468 20241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:23.801540 20241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:23.801585 20241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:23.810964 20241 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:34137
I20260927 10:52:23.810988 20369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:34137 every 8 connection(s)
I20260927 10:52:23.811928 20241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:52:23.820729 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 20241
I20260927 10:52:23.820837 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
Sep 27 10:52:23 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506343, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:23.821174 6270 external_mini_cluster.cc:1428] Setting key 0c3ef55c7949a874fab98b8016c6431e
I20260927 10:52:23.825294 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:23.813709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:59837 (local address 127.6.31.190:36581)
0927 10:52:23.813883 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:23.813888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:23.814880 (+ 992us) server_negotiation.cc:407] Connection header received
0927 10:52:23.815799 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:23.815801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:23.815848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:23.815924 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:23.817083 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.817619 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.818714 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.818940 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.819022 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:23.821554 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:23.821558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:23.821561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:23.821599 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:23.823263 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:23.823764 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:23.823770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:23.823774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:23.823862 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:23.824205 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:23.824208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:23.824210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:23.824401 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:23.824507 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:23.825003 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 10:52:23.825125 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:52:23.826380 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:23.814087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:59837)
0927 10:52:23.814731 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:23.814772 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:23.815604 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:23.816102 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:23.816108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:23.816476 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:23.816956 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:23.816968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.817720 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.817722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:23.818598 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:23.818606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:23.819069 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:23.819071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:23.819278 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:23.819820 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:23.819836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:23.821385 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:23.823377 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:23.823383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:23.823397 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:23.823657 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:23.823959 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:23.823962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:23.823964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:23.824102 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:23.824506 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:23.824513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:23.824766 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 10:52:23.824960 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 10:52:23.825280 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":113,"threads_started":1}
I20260927 10:52:23.827744 20370 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36581
I20260927 10:52:23.828019 20370 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:23.828521 20370 heartbeater.cc:511] Master 127.6.31.190:36581 requested a full tablet report, sending...
I20260927 10:52:23.829646 20045 ts_manager.cc:194] Registered new tserver with Master: 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
I20260927 10:52:23.830317 20045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:59837
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:52:23.850080 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36581
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:23.959618 20375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:23.959908 20375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:23.960014 20375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:23.964125 20375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:23.964208 20375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:23.964303 20375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:52:23.969528 20375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:23.970585 20375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:23.971644 20375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:23.981091 20380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:23.981122 20381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:23.981204 20375 server_base.cc:1034] running on GCE node
W20260927 10:52:23.981105 20383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:23.981757 20375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:23.982432 20375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:23.983637 20375 hybrid_clock.cc:648] HybridClock initialized: now 1790506343983611 us; error 45 us; skew 500 ppm
Sep 27 10:52:23 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506343, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:23.986405 20375 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:52:23.987314 20375 webserver.cc:460] Webserver started at http://127.6.31.131:45523/ using document root <none> and password file <none>
I20260927 10:52:23.987866 20375 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:23.987958 20375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:23.988255 20375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:23.990312 20375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "db33307922e7416480614c958b045723"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "fbca3640156b8a97521ddbb5a7133808"
server_key_iv: "3a496d9e3cc99d419af1a36f09cb6ec0"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.990871 20375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "db33307922e7416480614c958b045723"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "fbca3640156b8a97521ddbb5a7133808"
server_key_iv: "3a496d9e3cc99d419af1a36f09cb6ec0"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.994585 20375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:52:23.997025 20390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:23.998181 20375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:52:23.998309 20375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "db33307922e7416480614c958b045723"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "fbca3640156b8a97521ddbb5a7133808"
server_key_iv: "3a496d9e3cc99d419af1a36f09cb6ec0"
server_key_version: "encryptionkey@0"
I20260927 10:52:23.998415 20375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:24.015205 20375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:24.018373 20375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:24.018576 20375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:24.019214 20375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:24.020272 20375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:24.020362 20375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:24.020434 20375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:24.020488 20375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:24.030622 20375 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:45341
I20260927 10:52:24.030712 20503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:45341 every 8 connection(s)
I20260927 10:52:24.031798 20375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:52:24.036226 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 20375
I20260927 10:52:24.036383 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:52:24.036698 6270 external_mini_cluster.cc:1428] Setting key d1e01c6a3f41a0bd7837f19f8d391222
Sep 27 10:52:24 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506343, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:24.046492 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.033772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:42099 (local address 127.6.31.190:36581)
0927 10:52:24.033941 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.033947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.034810 (+ 863us) server_negotiation.cc:407] Connection header received
0927 10:52:24.035674 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.035677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.035722 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.035822 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:24.037098 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.037908 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.039334 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.039541 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.039620 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.042413 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:24.042418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:24.042421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:24.042455 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:24.044412 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.045059 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.045062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.045063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.045140 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.045416 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.045418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.045419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.045695 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:24.045845 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.046263 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.046368 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:52:24.047518 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.033999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:42099)
0927 10:52:24.034626 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.034711 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:24.035503 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:24.035961 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:24.035967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:24.036296 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:24.036939 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.036953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.038051 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.038055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.039235 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.039244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.039687 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.039690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.039859 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.040502 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:24.040518 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:24.042216 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:24.044559 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:24.044568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:24.044587 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:24.044964 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:24.045205 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:24.045207 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:24.045208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:24.045340 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:24.045789 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:24.045795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:24.046046 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 10:52:24.046206 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 10:52:24.046478 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":109,"threads_started":1}
I20260927 10:52:24.048714 20504 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36581
I20260927 10:52:24.048947 20504 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:24.049544 20504 heartbeater.cc:511] Master 127.6.31.190:36581 requested a full tablet report, sending...
I20260927 10:52:24.050585 20045 ts_manager.cc:194] Registered new tserver with Master: db33307922e7416480614c958b045723 (127.6.31.131:45341)
I20260927 10:52:24.051223 20045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:42099
I20260927 10:52:24.062444 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:52:24.071882 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.064479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.6.31.190:36581)
0927 10:52:24.064620 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.064623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.064769 (+ 146us) server_negotiation.cc:407] Connection header received
0927 10:52:24.064865 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.064867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.064908 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.064976 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:24.065660 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.066263 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.067044 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.067319 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.067398 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.068664 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:24.068667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:24.068669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:24.068690 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:24.070055 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.070494 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.070496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.070497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.070567 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.070878 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.070880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.070882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.071153 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:24.071266 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.071515 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.071610 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:52:24.074783 20045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:52:24.077859 20045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:52:24.092322 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.086396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.6.31.130:34137)
0927 10:52:24.087076 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.087113 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.087136 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:52:24.087613 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.087620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.087904 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.088058 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:24.088365 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.089358 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.090940 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.091590 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.091769 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.091921 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.092067 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.092163 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":115,"threads_started":1}
I20260927 10:52:24.092358 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.086288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34137 (local address 127.0.0.1:38716)
0927 10:52:24.086741 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.086793 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:24.086926 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:24.088058 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:24.088064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:24.088085 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:24.088239 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.088247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.089548 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.089551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.090819 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.090828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.091757 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.091761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.091875 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.091912 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 10:52:24.092028 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:52:24.092159 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":5,"thread_start_us":99,"threads_started":1}
I20260927 10:52:24.093238 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.087070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:34075 (local address 127.0.0.1:39218)
0927 10:52:24.087675 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.087689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:24.087769 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:24.088484 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:24.088487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:24.088505 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:24.088591 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.088597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.090159 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.090162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.091702 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.091712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.092836 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.092840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.092936 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.092958 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:52:24.093034 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:52:24.093099 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":111,"thread_start_us":108,"threads_started":1}
I20260927 10:52:24.093490 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.086391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:45341 (local address 127.0.0.1:44860)
0927 10:52:24.087226 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.087242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:24.087332 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:24.088215 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:24.088218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:24.088237 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:24.088375 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.088383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.090159 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.090162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.091702 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.091712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.093066 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.093070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.093166 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.093189 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:52:24.093263 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:52:24.093361 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":217,"thread_start_us":91,"threads_started":1}
I20260927 10:52:24.093655 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.087284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.6.31.131:45341)
0927 10:52:24.087729 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.087744 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.087768 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:52:24.087850 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.087855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.088051 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.088523 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:24.088688 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.089985 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.091852 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.092902 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.093051 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.093177 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.093414 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.093511 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":133,"threads_started":1}
I20260927 10:52:24.094166 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.087340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.6.31.129:34075)
0927 10:52:24.087965 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.087981 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.088001 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:52:24.088074 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.088079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.088270 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.088505 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:24.088682 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.090025 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.091847 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.092717 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.092825 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.092963 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.093978 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.094058 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":109,"threads_started":1}
I20260927 10:52:24.095846 20304 tablet_service.cc:1467] Processing CreateTablet for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 (DEFAULT_TABLE table=test-table [id=eb900b1d2bd14c5a9736dbb6e1413c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:24.096091 20438 tablet_service.cc:1467] Processing CreateTablet for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 (DEFAULT_TABLE table=test-table [id=eb900b1d2bd14c5a9736dbb6e1413c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:24.096349 20170 tablet_service.cc:1467] Processing CreateTablet for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 (DEFAULT_TABLE table=test-table [id=eb900b1d2bd14c5a9736dbb6e1413c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:24.097014 20438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:24.097009 20304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:24.097266 20170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:24.103974 20521 tablet_bootstrap.cc:492] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Bootstrap starting.
I20260927 10:52:24.104527 20523 tablet_bootstrap.cc:492] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: Bootstrap starting.
I20260927 10:52:24.106266 20521 tablet_bootstrap.cc:654] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:24.107005 20523 tablet_bootstrap.cc:654] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:24.107630 20521 log.cc:824] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:24.107761 20523 log.cc:824] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:24.108108 20522 tablet_bootstrap.cc:492] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: Bootstrap starting.
I20260927 10:52:24.109678 20523 tablet_bootstrap.cc:492] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: No bootstrap required, opened a new log
I20260927 10:52:24.109787 20521 tablet_bootstrap.cc:492] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: No bootstrap required, opened a new log
I20260927 10:52:24.109920 20523 ts_tablet_manager.cc:1397] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:52:24.109969 20521 ts_tablet_manager.cc:1397] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:52:24.110713 20522 tablet_bootstrap.cc:654] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:24.111572 20522 log.cc:824] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:24.113117 20523 raft_consensus.cc:348] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.113361 20523 raft_consensus.cc:374] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:24.113413 20523 raft_consensus.cc:729] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374a36d475454ef78c4ad803b22f0158, State: Initialized, Role: FOLLOWER
I20260927 10:52:24.113605 20522 tablet_bootstrap.cc:492] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: No bootstrap required, opened a new log
I20260927 10:52:24.113799 20522 ts_tablet_manager.cc:1397] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:52:24.113706 20521 raft_consensus.cc:348] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.113925 20521 raft_consensus.cc:374] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:24.113867 20523 consensus_queue.cc:260] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.113978 20521 raft_consensus.cc:729] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42766c759ce44956a5ffcd71e43f36b1, State: Initialized, Role: FOLLOWER
I20260927 10:52:24.114431 20521 consensus_queue.cc:260] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.114851 20523 ts_tablet_manager.cc:1428] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260927 10:52:24.115257 20370 heartbeater.cc:503] Master 127.6.31.190:36581 was elected leader, sending a full tablet report...
I20260927 10:52:24.115347 20521 ts_tablet_manager.cc:1428] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 10:52:24.115628 20236 heartbeater.cc:503] Master 127.6.31.190:36581 was elected leader, sending a full tablet report...
I20260927 10:52:24.118263 20522 raft_consensus.cc:348] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.118515 20522 raft_consensus.cc:374] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:24.118568 20522 raft_consensus.cc:729] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db33307922e7416480614c958b045723, State: Initialized, Role: FOLLOWER
I20260927 10:52:24.118993 20522 consensus_queue.cc:260] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.119633 20504 heartbeater.cc:503] Master 127.6.31.190:36581 was elected leader, sending a full tablet report...
I20260927 10:52:24.119939 20522 ts_tablet_manager.cc:1428] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:52:24.134068 20527 raft_consensus.cc:482] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:24.134235 20527 raft_consensus.cc:504] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.135360 20527 leader_election.cc:288] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 42766c759ce44956a5ffcd71e43f36b1 (127.6.31.129:34075)
I20260927 10:52:24.139354 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.135667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:44569 (local address 127.6.31.131:45341)
0927 10:52:24.135810 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.135815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.135830 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:52:24.135984 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.135991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.136050 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.136133 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:24.136407 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.136924 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.137957 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.138664 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.138775 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.138835 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.139152 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.139223 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:52:24.139361 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.135545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:45341 (local address 127.6.31.130:44569)
0927 10:52:24.135724 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.135742 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:24.135837 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:24.136151 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:24.136155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:24.136172 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:24.136302 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.136307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.137041 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.137045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.137814 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.137822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.138892 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.138895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.139003 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.139025 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:52:24.139134 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:52:24.139205 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 10:52:24.139979 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.135998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:59037 (local address 127.6.31.129:34075)
0927 10:52:24.136124 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.136129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.136304 (+ 175us) server_negotiation.cc:407] Connection header received
0927 10:52:24.136423 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.136426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.136476 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.136557 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:24.136878 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.137362 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.138639 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.139266 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.139387 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.139439 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.139777 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.139866 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:52:24.139969 20458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8eaf0ff0ac482ea6c352ce6dc682c3" candidate_uuid: "374a36d475454ef78c4ad803b22f0158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33307922e7416480614c958b045723" is_pre_election: true
I20260927 10:52:24.140249 20458 raft_consensus.cc:2454] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 374a36d475454ef78c4ad803b22f0158 in term 0.
I20260927 10:52:24.140311 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.135891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:34075 (local address 127.6.31.130:59037)
0927 10:52:24.136218 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.136228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:24.136335 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:24.136569 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:24.136571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:24.136585 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:24.136763 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.136768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.137472 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.137477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.138486 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.138500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.139529 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.139536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.139634 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.139656 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:52:24.140103 (+ 447us) client_negotiation.cc:240] Negotiation successful
0927 10:52:24.140180 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260927 10:52:24.140719 20260 leader_election.cc:302] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 374a36d475454ef78c4ad803b22f0158, db33307922e7416480614c958b045723; no voters:
I20260927 10:52:24.140868 20190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8eaf0ff0ac482ea6c352ce6dc682c3" candidate_uuid: "374a36d475454ef78c4ad803b22f0158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42766c759ce44956a5ffcd71e43f36b1" is_pre_election: true
I20260927 10:52:24.141004 20527 raft_consensus.cc:2781] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:52:24.141098 20527 raft_consensus.cc:482] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:52:24.141149 20527 raft_consensus.cc:3037] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:24.141136 20190 raft_consensus.cc:2454] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 374a36d475454ef78c4ad803b22f0158 in term 0.
I20260927 10:52:24.142192 20527 raft_consensus.cc:504] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.142467 20527 leader_election.cc:288] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [CANDIDATE]: Term 1 election: Requested vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 42766c759ce44956a5ffcd71e43f36b1 (127.6.31.129:34075)
I20260927 10:52:24.142802 20458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8eaf0ff0ac482ea6c352ce6dc682c3" candidate_uuid: "374a36d475454ef78c4ad803b22f0158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33307922e7416480614c958b045723"
I20260927 10:52:24.142892 20458 raft_consensus.cc:3037] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:24.142904 20190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8eaf0ff0ac482ea6c352ce6dc682c3" candidate_uuid: "374a36d475454ef78c4ad803b22f0158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42766c759ce44956a5ffcd71e43f36b1"
I20260927 10:52:24.143028 20190 raft_consensus.cc:3037] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:24.143889 20458 raft_consensus.cc:2454] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 374a36d475454ef78c4ad803b22f0158 in term 1.
I20260927 10:52:24.144084 20190 raft_consensus.cc:2454] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 374a36d475454ef78c4ad803b22f0158 in term 1.
I20260927 10:52:24.144148 20260 leader_election.cc:302] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 374a36d475454ef78c4ad803b22f0158, db33307922e7416480614c958b045723; no voters:
I20260927 10:52:24.144294 20527 raft_consensus.cc:2781] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:52:24.144515 20527 raft_consensus.cc:686] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 1 LEADER]: Becoming Leader. State: Replica: 374a36d475454ef78c4ad803b22f0158, State: Running, Role: LEADER
I20260927 10:52:24.144867 20527 consensus_queue.cc:237] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.148180 20045 catalog_manager.cc:5495] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 reported cstate change: term changed from 0 to 1, leader changed from <none> to 374a36d475454ef78c4ad803b22f0158 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "374a36d475454ef78c4ad803b22f0158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } health_report { overall_health: UNKNOWN } } }
I20260927 10:52:24.160246 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.156807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.6.31.130:34137)
0927 10:52:24.156928 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.156931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.156943 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:52:24.157025 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.157027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.157078 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.157167 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:52:24.157472 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.157988 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.158812 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.159020 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.159165 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.159496 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:52:24.159920 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:52:24.160027 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.160088 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.160136 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:52:24.167289 20458 raft_consensus.cc:1261] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [term 1 FOLLOWER]: Refusing update from remote peer 374a36d475454ef78c4ad803b22f0158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:24.167286 20190 raft_consensus.cc:1261] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Refusing update from remote peer 374a36d475454ef78c4ad803b22f0158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:24.167982 20527 consensus_queue.cc:1035] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:24.168286 20531 consensus_queue.cc:1035] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:24.176333 20535 mvcc.cc:205] Tried to move back new op lower bound from 7333913985701466112 to 7333913985620148224. Current Snapshot: MvccSnapshot[applied={T|T < 7333913985701466112}]
I20260927 10:52:24.185336 20045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:52:24.185531 20045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:52:24.188010 20045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Sending DeleteTablet for 3 replicas of tablet cc8eaf0ff0ac482ea6c352ce6dc682c3
I20260927 10:52:24.188931 20170 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:24 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:39218
I20260927 10:52:24.188978 20438 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:24 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:44860
I20260927 10:52:24.188972 20304 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:24 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:38716
I20260927 10:52:24.189555 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 20107
I20260927 10:52:24.195344 20546 tablet_replica.cc:330] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: stopping tablet replica
I20260927 10:52:24.195710 20546 raft_consensus.cc:2229] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:52:24.196110 20545 tablet_replica.cc:330] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: stopping tablet replica
I20260927 10:52:24.196151 20546 raft_consensus.cc:2258] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:52:24.196692 20545 raft_consensus.cc:2229] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:52:24.197052 20545 raft_consensus.cc:2258] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:52:24.198064 20546 ts_tablet_manager.cc:1905] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:52:24.198540 20545 ts_tablet_manager.cc:1905] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:52:24.200627 20032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:34075 (error 108)
I20260927 10:52:24.200853 20546 ts_tablet_manager.cc:1918] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:52:24.200963 20546 log.cc:1196] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/cc8eaf0ff0ac482ea6c352ce6dc682c3
I20260927 10:52:24.201326 20546 ts_tablet_manager.cc:1939] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 374a36d475454ef78c4ad803b22f0158: Deleting consensus metadata
I20260927 10:52:24.201448 20545 ts_tablet_manager.cc:1918] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:52:24.201578 20545 log.cc:1196] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/cc8eaf0ff0ac482ea6c352ce6dc682c3
I20260927 10:52:24.201939 20545 ts_tablet_manager.cc:1939] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P db33307922e7416480614c958b045723: Deleting consensus metadata
I20260927 10:52:24.202265 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.201902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:34075 (local address 127.0.0.1:39224)
0927 10:52:24.202093 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.202161 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:34075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260927 10:52:24.202504 20032 catalog_manager.cc:4584] TS 42766c759ce44956a5ffcd71e43f36b1 (127.6.31.129:34075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3: Network error: Client connection negotiation failed: client connection to 127.6.31.129:34075: connect: Connection refused (error 111)
I20260927 10:52:24.202586 20031 catalog_manager.cc:4841] TS 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137): tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 (table test-table [id=eb900b1d2bd14c5a9736dbb6e1413c94]) successfully deleted
I20260927 10:52:24.202836 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 20241
I20260927 10:52:24.203550 20032 catalog_manager.cc:4841] TS db33307922e7416480614c958b045723 (127.6.31.131:45341): tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 (table test-table [id=eb900b1d2bd14c5a9736dbb6e1413c94]) successfully deleted
I20260927 10:52:24.208595 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 20375
I20260927 10:52:24.214229 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 20013
I20260927 10:52:24.220525 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36581
--webserver_interface=127.6.31.190
--webserver_port=44665
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:36581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:24.332121 20548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:24.332449 20548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:24.332532 20548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:24.336661 20548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:24.336745 20548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:24.336781 20548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:24.336820 20548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:24.336855 20548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:24.342242 20548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:36581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=44665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:24.343371 20548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:24.344425 20548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:52:24.357137 20548 server_base.cc:1034] running on GCE node
W20260927 10:52:24.357079 20555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:24.357082 20554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:24.357079 20557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:24.357949 20548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:24.359195 20548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:24.360401 20548 hybrid_clock.cc:648] HybridClock initialized: now 1790506344360385 us; error 34 us; skew 500 ppm
Sep 27 10:52:24 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506344, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:24.363381 20548 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:52:24.364451 20548 webserver.cc:460] Webserver started at http://127.6.31.190:44665/ using document root <none> and password file <none>
I20260927 10:52:24.365082 20548 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:24.365188 20548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:24.369141 20548 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:52:24.371358 20564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:24.372516 20548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:52:24.372669 20548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "8976e07ff09c4e16b58dbc2b512a4504"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "333f1f36aed65d93211b3493c8ee588e"
server_key_iv: "05f066dae6ef62a9b10e9404593f9df9"
server_key_version: "encryptionkey@0"
I20260927 10:52:24.373265 20548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:24.395339 20548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:24.398628 20548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:24.398852 20548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:24.406068 20548 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:36581
I20260927 10:52:24.406286 20616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:36581 every 8 connection(s)
I20260927 10:52:24.407298 20548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:52:24.411592 20617 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:52:24.413115 20617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Bootstrap starting.
I20260927 10:52:24.416752 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 20548
I20260927 10:52:24.417169 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:34075
--local_ip_for_outbound_sockets=127.6.31.129
--tserver_master_addrs=127.6.31.190:36581
--webserver_port=38395
--webserver_interface=127.6.31.129
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:52:24.433921 20617 log.cc:824] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:24.445191 20617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:52:24.445602 20617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Bootstrap complete.
I20260927 10:52:24.450963 20617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:24.451493 20617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8976e07ff09c4e16b58dbc2b512a4504, State: Initialized, Role: FOLLOWER
I20260927 10:52:24.452042 20617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:24.452189 20617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:52:24.452260 20617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:52:24.452353 20617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:52:24.454680 20617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:24.455040 20617 leader_election.cc:302] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8976e07ff09c4e16b58dbc2b512a4504; no voters:
I20260927 10:52:24.455406 20617 leader_election.cc:288] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:52:24.455551 20621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:52:24.455875 20621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 2 LEADER]: Becoming Leader. State: Replica: 8976e07ff09c4e16b58dbc2b512a4504, State: Running, Role: LEADER
I20260927 10:52:24.456189 20621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:24.456646 20617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:52:24.457665 20623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8976e07ff09c4e16b58dbc2b512a4504. Latest consensus state: current_term: 2 leader_uuid: "8976e07ff09c4e16b58dbc2b512a4504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } } }
I20260927 10:52:24.457665 20622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8976e07ff09c4e16b58dbc2b512a4504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } } }
I20260927 10:52:24.457813 20623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:24.457813 20622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:24.458528 20631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:52:24.463264 20631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:52:24.463898 20631 catalog_manager.cc:1259] Loaded cluster ID: 1ede30254f5d42a6987e58824736f3c1
I20260927 10:52:24.463955 20631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:52:24.465355 20631 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 10:52:24.470283 20631 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 10:52:24.475924 20631 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:52:24.476728 20631 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Loaded TSK: 0
I20260927 10:52:24.477917 20631 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:52:24.541172 20619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:24.541486 20619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:24.541561 20619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:24.545682 20619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:24.545774 20619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:24.545868 20619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:52:24.551265 20619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:34075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=38395
--tserver_master_addrs=127.6.31.190:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:24.552274 20619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:24.553396 20619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:24.564496 20645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:24.564535 20647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:24.564477 20644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:24.565124 20619 server_base.cc:1034] running on GCE node
I20260927 10:52:24.565610 20619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:24.566274 20619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:24.567459 20619 hybrid_clock.cc:648] HybridClock initialized: now 1790506344567443 us; error 49 us; skew 500 ppm
Sep 27 10:52:24 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506344, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:24.570248 20619 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:52:24.571280 20619 webserver.cc:460] Webserver started at http://127.6.31.129:38395/ using document root <none> and password file <none>
I20260927 10:52:24.571823 20619 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:24.571894 20619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:24.575986 20619 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 10:52:24.578236 20654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:24.579571 20619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:52:24.579661 20619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "42766c759ce44956a5ffcd71e43f36b1"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "34c10f8c1001f0b5860a264ae7b1865d"
server_key_iv: "e9b5546ecec134278e7251c2eb13f335"
server_key_version: "encryptionkey@0"
I20260927 10:52:24.580216 20619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:24.601390 20619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:24.604641 20619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:24.604854 20619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:24.605491 20619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:24.606570 20661 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:52:24.610951 20619 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:52:24.611008 20619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 10:52:24.611079 20619 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:52:24.612655 20619 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:52:24.612697 20619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:52:24.612977 20661 tablet_bootstrap.cc:492] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Bootstrap starting.
I20260927 10:52:24.623040 20619 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:34075
I20260927 10:52:24.623159 20768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:34075 every 8 connection(s)
I20260927 10:52:24.624030 20619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:52:24.624846 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 20619
I20260927 10:52:24.625361 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:34137
--local_ip_for_outbound_sockets=127.6.31.130
--tserver_master_addrs=127.6.31.190:36581
--webserver_port=35975
--webserver_interface=127.6.31.130
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:52:24.633976 20661 log.cc:824] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:52:24 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506344, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:24.641588 20661 tablet_bootstrap.cc:492] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:52:24.641988 20661 tablet_bootstrap.cc:492] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Bootstrap complete.
I20260927 10:52:24.643196 20661 ts_tablet_manager.cc:1397] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.015s
I20260927 10:52:24.644174 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.626291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:58203)
0927 10:52:24.627554 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.627612 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:24.628676 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:24.630061 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:24.630067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:24.630435 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:24.630883 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.630906 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.632727 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.632734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.633680 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.633690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.634615 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.634623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.634869 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.635653 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:24.635684 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:24.637818 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:24.640716 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:24.640725 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:24.640753 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:24.641096 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:24.641492 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:24.641495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:24.641497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:24.641655 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:24.642367 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:24.642373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:24.642641 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 10:52:24.642906 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 10:52:24.643250 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":153,"threads_started":1}
I20260927 10:52:24.644663 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.626319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:58203 (local address 127.6.31.190:36581)
0927 10:52:24.626825 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.626846 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.627779 (+ 933us) server_negotiation.cc:407] Connection header received
0927 10:52:24.629125 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.629140 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.629574 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.630029 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:24.631108 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.632535 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.633858 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.634454 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.634645 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.638077 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:24.638086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:24.638106 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:24.638143 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:24.640558 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.641230 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.641236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.641242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.641386 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.641770 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.641774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.641776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.642190 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:24.642313 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.643192 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.643559 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":128,"threads_started":1}
I20260927 10:52:24.646024 20661 raft_consensus.cc:348] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.646441 20661 raft_consensus.cc:729] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42766c759ce44956a5ffcd71e43f36b1, State: Initialized, Role: FOLLOWER
I20260927 10:52:24.646924 20661 consensus_queue.cc:260] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:24.647104 20769 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36581
I20260927 10:52:24.647325 20769 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:24.647873 20769 heartbeater.cc:511] Master 127.6.31.190:36581 requested a full tablet report, sending...
I20260927 10:52:24.647900 20661 ts_tablet_manager.cc:1428] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:52:24.650444 20581 ts_manager.cc:194] Registered new tserver with Master: 42766c759ce44956a5ffcd71e43f36b1 (127.6.31.129:34075)
I20260927 10:52:24.653934 20581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:58203
I20260927 10:52:24.656337 20769 heartbeater.cc:503] Master 127.6.31.190:36581 was elected leader, sending a full tablet report...
Sep 27 10:52:24 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506344, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:52:24.664990 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.653377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.6.31.129:34075)
0927 10:52:24.653774 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.653785 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.654182 (+ 397us) server_negotiation.cc:407] Connection header received
0927 10:52:24.654454 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.654460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.654657 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.654774 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:24.655692 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.656720 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.658053 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.658430 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.658558 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.661200 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:24.661208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:24.661218 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:24.661240 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:24.662998 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.663663 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.663668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.663669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.663747 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.664027 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.664029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.664030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.664269 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:24.664367 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.664730 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.664865 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":114,"threads_started":1}
I20260927 10:52:24.665021 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.653267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:34075 (local address 127.0.0.1:39236)
0927 10:52:24.654046 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.654094 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:24.654290 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:24.654975 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:24.654981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:24.655430 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:24.655554 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.655561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.656920 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.656924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.657876 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.657887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.658569 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.658573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.658681 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.659333 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:24.659348 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:24.661057 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:24.663127 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:24.663137 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:24.663142 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:24.663526 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:24.663847 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:24.663850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:24.663852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:24.663946 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:24.664377 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:24.664383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:24.664509 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 10:52:24.664644 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 10:52:24.664846 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":113,"threads_started":1}
I20260927 10:52:24.666072 20703 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:24 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:39236
I20260927 10:52:24.666947 20778 tablet_replica.cc:330] stopping tablet replica
I20260927 10:52:24.667342 20778 raft_consensus.cc:2229] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:52:24.667619 20778 raft_consensus.cc:2258] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:52:24.668725 20778 ts_tablet_manager.cc:1905] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:52:24.673332 20778 ts_tablet_manager.cc:1918] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:52:24.673444 20778 log.cc:1196] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/cc8eaf0ff0ac482ea6c352ce6dc682c3
I20260927 10:52:24.673720 20778 ts_tablet_manager.cc:1939] T cc8eaf0ff0ac482ea6c352ce6dc682c3 P 42766c759ce44956a5ffcd71e43f36b1: Deleting consensus metadata
I20260927 10:52:24.675817 20568 catalog_manager.cc:4841] TS 42766c759ce44956a5ffcd71e43f36b1 (127.6.31.129:34075): tablet cc8eaf0ff0ac482ea6c352ce6dc682c3 (table test-table [id=eb900b1d2bd14c5a9736dbb6e1413c94]) successfully deleted
W20260927 10:52:24.752969 20773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:24.753252 20773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:24.753296 20773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:24.758205 20773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:24.758273 20773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:24.758335 20773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:52:24.763932 20773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:34137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=35975
--tserver_master_addrs=127.6.31.190:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:24.764964 20773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:24.766049 20773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:24.776036 20786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:24.776018 20784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:24.776005 20783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:24.776396 20773 server_base.cc:1034] running on GCE node
I20260927 10:52:24.776829 20773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:24.777386 20773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:24.778582 20773 hybrid_clock.cc:648] HybridClock initialized: now 1790506344778543 us; error 63 us; skew 500 ppm
Sep 27 10:52:24 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506344, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:24.781291 20773 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:52:24.782084 20773 webserver.cc:460] Webserver started at http://127.6.31.130:35975/ using document root <none> and password file <none>
I20260927 10:52:24.782586 20773 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:24.782660 20773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:24.786614 20773 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:52:24.788733 20793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:24.789937 20773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:52:24.790055 20773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "374a36d475454ef78c4ad803b22f0158"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "2614df765363825ed093a1aa3cec6934"
server_key_iv: "6dad4089fb1fc9d248af6d62a80e0bd1"
server_key_version: "encryptionkey@0"
I20260927 10:52:24.790580 20773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:24.800930 20773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:24.804207 20773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:24.804415 20773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:24.805015 20773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:24.805846 20773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:24.805892 20773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:24.805960 20773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:24.805994 20773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:24.815052 20773 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:34137
I20260927 10:52:24.815142 20906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:34137 every 8 connection(s)
I20260927 10:52:24.816171 20773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:52:24 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506344, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:24.826087 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 20773
I20260927 10:52:24.826511 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:45341
--local_ip_for_outbound_sockets=127.6.31.131
--tserver_master_addrs=127.6.31.190:36581
--webserver_port=45523
--webserver_interface=127.6.31.131
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:52:24.829747 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.818044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:47935 (local address 127.6.31.190:36581)
0927 10:52:24.818186 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:24.818190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:24.819178 (+ 988us) server_negotiation.cc:407] Connection header received
0927 10:52:24.820114 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:24.820117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:24.820179 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:24.820285 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:24.821455 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.822000 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.823222 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.823463 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.823576 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.826034 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:24.826037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:24.826039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:24.826065 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:24.827650 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.828185 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.828189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.828191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.828281 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:24.828620 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:24.828625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:24.828627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:24.828825 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:24.828932 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:24.829435 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 10:52:24.829572 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:52:24.830660 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:24.818365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:47935)
0927 10:52:24.819013 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:24.819052 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:24.819894 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:24.820473 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:24.820480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:24.820831 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:24.821308 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.821321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.822128 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.822131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.823044 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:24.823054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:24.823617 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:24.823621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:24.823796 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:24.824415 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:24.824431 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:24.825851 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:24.827797 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:24.827803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:24.827818 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:24.828077 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:24.828380 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:24.828383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:24.828385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:24.828524 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:24.828931 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:24.828938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:24.829185 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 10:52:24.829379 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 10:52:24.829701 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":117,"threads_started":1}
I20260927 10:52:24.831900 20907 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36581
I20260927 10:52:24.832192 20907 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:24.832828 20907 heartbeater.cc:511] Master 127.6.31.190:36581 requested a full tablet report, sending...
I20260927 10:52:24.834080 20581 ts_manager.cc:194] Registered new tserver with Master: 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
I20260927 10:52:24.834764 20581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:47935
W20260927 10:52:24.937463 20910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:24.937718 20910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:24.937760 20910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:24.941771 20910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:24.941828 20910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:24.941917 20910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:52:24.947033 20910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:45341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=45523
--tserver_master_addrs=127.6.31.190:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:24.948056 20910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:24.949030 20910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:24.958534 20916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:24.958534 20918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:24.958636 20915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:24.958930 20910 server_base.cc:1034] running on GCE node
I20260927 10:52:24.959359 20910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:24.959995 20910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:24.961158 20910 hybrid_clock.cc:648] HybridClock initialized: now 1790506344961135 us; error 32 us; skew 500 ppm
Sep 27 10:52:24 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506344, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:24.963994 20910 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:52:24.964880 20910 webserver.cc:460] Webserver started at http://127.6.31.131:45523/ using document root <none> and password file <none>
I20260927 10:52:24.965435 20910 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:24.965505 20910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:24.969218 20910 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:52:24.971313 20925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:24.972501 20910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:52:24.972656 20910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "db33307922e7416480614c958b045723"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "fbca3640156b8a97521ddbb5a7133808"
server_key_iv: "3a496d9e3cc99d419af1a36f09cb6ec0"
server_key_version: "encryptionkey@0"
I20260927 10:52:24.973179 20910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:24.995229 20910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:24.998623 20910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:24.998844 20910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:24.999511 20910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:25.000346 20910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:25.000416 20910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:25.000483 20910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:25.000512 20910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:25.009661 20910 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:45341
I20260927 10:52:25.009685 21038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:45341 every 8 connection(s)
I20260927 10:52:25.010700 20910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:52:25.013306 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 20910
Sep 27 10:52:25 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506344, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:25.022948 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.014024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.6.31.190:36581)
0927 10:52:25.014306 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:25.014311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:25.014332 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:52:25.014469 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:25.014472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:25.014521 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:25.014612 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:25.015513 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.016308 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.017286 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.017510 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.017586 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.019157 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:25.019163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:25.019166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:25.019239 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:25.021067 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:25.021487 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:25.021492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:25.021495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:25.021583 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:25.021875 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:25.021880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:25.021885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:25.022142 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:25.022295 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:25.022567 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 10:52:25.022710 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
I20260927 10:52:25.026000 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.012605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:38863 (local address 127.6.31.190:36581)
0927 10:52:25.012745 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:25.012748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:25.013600 (+ 852us) server_negotiation.cc:407] Connection header received
0927 10:52:25.015249 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:25.015260 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:25.015312 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:25.015407 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:25.016939 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.017533 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.018726 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.019077 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.019180 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.022132 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:25.022134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:25.022136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:25.022157 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:25.023990 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:25.024473 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:25.024476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:25.024478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:25.024586 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:25.025057 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:25.025059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:25.025060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:25.025210 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:25.025288 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:25.025795 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 10:52:25.025892 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55,"spinlock_wait_cycles":18176}
I20260927 10:52:25.026760 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.012974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:38863)
0927 10:52:25.013425 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:25.013488 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:25.015005 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:25.015667 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:25.015688 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:25.016153 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:25.016801 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.016818 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.017752 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.017755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.018596 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.018605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.019229 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.019233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.019436 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.020169 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:25.020192 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:25.021956 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:25.024109 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:25.024114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:25.024126 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:25.024374 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:25.024728 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:25.024731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:25.024733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:25.024951 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:25.025321 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:25.025326 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:25.025545 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 10:52:25.025698 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 10:52:25.025952 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":132,"threads_started":1}
I20260927 10:52:25.027889 21039 heartbeater.cc:347] Connected to a master server at 127.6.31.190:36581
I20260927 10:52:25.028136 21039 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:25.028605 21039 heartbeater.cc:511] Master 127.6.31.190:36581 requested a full tablet report, sending...
I20260927 10:52:25.029628 20581 ts_manager.cc:194] Registered new tserver with Master: db33307922e7416480614c958b045723 (127.6.31.131:45341)
I20260927 10:52:25.030174 20581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:38863
I20260927 10:52:25.036499 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:52:25.040789 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.037043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.6.31.190:36581)
0927 10:52:25.037173 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:25.037179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:25.037373 (+ 194us) server_negotiation.cc:407] Connection header received
0927 10:52:25.037552 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:25.037555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:25.037655 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:25.037762 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:52:25.038000 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.038516 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.039444 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.039678 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.039788 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.040085 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:52:25.040431 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:52:25.040559 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:25.040620 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 10:52:25.040666 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:52:25.041327 20581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:52:25.044483 20581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:52:25.052822 20703 tablet_service.cc:1467] Processing CreateTablet for tablet 51cdc5556e474b9296f482ce7bdbfff7 (DEFAULT_TABLE table=test-table [id=1647c9c03a4c46b8a231516df7a6e35c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:25.053491 20703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cdc5556e474b9296f482ce7bdbfff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:25.056298 20661 tablet_bootstrap.cc:492] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Bootstrap starting.
I20260927 10:52:25.057269 20661 tablet_bootstrap.cc:654] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:25.058347 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.051782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:45341 (local address 127.0.0.1:44868)
0927 10:52:25.052073 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:25.052089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:25.052164 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:25.053338 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:25.053341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:25.053357 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:25.053460 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.053466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.055027 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.055031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.056398 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.056409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.057877 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.057880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.058013 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.058089 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 10:52:25.058165 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:52:25.058237 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":68}
I20260927 10:52:25.058506 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.052083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.6.31.131:45341)
0927 10:52:25.052858 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:25.052884 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:25.052904 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:52:25.052964 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:25.052968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:25.053199 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:25.053358 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:25.053644 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.054868 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.056895 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.057778 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.057873 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.058030 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:25.058222 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 10:52:25.058339 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":116,"threads_started":1}
I20260927 10:52:25.058807 20661 tablet_bootstrap.cc:492] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: No bootstrap required, opened a new log
I20260927 10:52:25.058887 20661 ts_tablet_manager.cc:1397] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 10:52:25.058990 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.052094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34137 (local address 127.0.0.1:38730)
0927 10:52:25.052467 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:25.052480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:25.052565 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:25.053531 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:25.053537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:25.053555 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:25.053638 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.053644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.055184 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.055187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.056514 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.056524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.058011 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.058015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.058684 (+ 669us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.058711 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:52:25.058783 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:52:25.058857 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":572,"thread_start_us":102,"threads_started":1}
I20260927 10:52:25.059430 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.052534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.6.31.130:34137)
0927 10:52:25.052895 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:25.052907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:25.052930 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:52:25.053008 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:25.053013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:25.053204 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:25.053359 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:25.053768 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.055001 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.056658 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.057606 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.057793 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.057958 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:25.059176 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0927 10:52:25.059289 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":126,"threads_started":1}
I20260927 10:52:25.059365 20661 raft_consensus.cc:348] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.059478 20661 raft_consensus.cc:374] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:25.059517 20661 raft_consensus.cc:729] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42766c759ce44956a5ffcd71e43f36b1, State: Initialized, Role: FOLLOWER
I20260927 10:52:25.059648 20661 consensus_queue.cc:260] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.059940 20661 ts_tablet_manager.cc:1428] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:52:25.061674 20973 tablet_service.cc:1467] Processing CreateTablet for tablet 51cdc5556e474b9296f482ce7bdbfff7 (DEFAULT_TABLE table=test-table [id=1647c9c03a4c46b8a231516df7a6e35c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:25.063149 20973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cdc5556e474b9296f482ce7bdbfff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:25.065161 20841 tablet_service.cc:1467] Processing CreateTablet for tablet 51cdc5556e474b9296f482ce7bdbfff7 (DEFAULT_TABLE table=test-table [id=1647c9c03a4c46b8a231516df7a6e35c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:25.066181 20841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cdc5556e474b9296f482ce7bdbfff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:25.071652 21052 tablet_bootstrap.cc:492] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: Bootstrap starting.
I20260927 10:52:25.073588 21053 tablet_bootstrap.cc:492] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: Bootstrap starting.
I20260927 10:52:25.073704 21052 tablet_bootstrap.cc:654] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:25.074698 21052 log.cc:824] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:25.075868 21053 tablet_bootstrap.cc:654] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:25.076592 21053 log.cc:824] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:25.076668 21052 tablet_bootstrap.cc:492] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: No bootstrap required, opened a new log
I20260927 10:52:25.076903 21052 ts_tablet_manager.cc:1397] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 10:52:25.078750 21053 tablet_bootstrap.cc:492] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: No bootstrap required, opened a new log
I20260927 10:52:25.078936 21053 ts_tablet_manager.cc:1397] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:52:25.081069 21052 raft_consensus.cc:348] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.081342 21052 raft_consensus.cc:374] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:25.081393 21052 raft_consensus.cc:729] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db33307922e7416480614c958b045723, State: Initialized, Role: FOLLOWER
I20260927 10:52:25.081838 21052 consensus_queue.cc:260] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.082542 21039 heartbeater.cc:503] Master 127.6.31.190:36581 was elected leader, sending a full tablet report...
I20260927 10:52:25.082856 21052 ts_tablet_manager.cc:1428] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260927 10:52:25.082702 21053 raft_consensus.cc:348] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.082943 21053 raft_consensus.cc:374] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:25.082995 21053 raft_consensus.cc:729] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374a36d475454ef78c4ad803b22f0158, State: Initialized, Role: FOLLOWER
I20260927 10:52:25.083413 21053 consensus_queue.cc:260] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.084266 21053 ts_tablet_manager.cc:1428] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:52:25.084586 20907 heartbeater.cc:503] Master 127.6.31.190:36581 was elected leader, sending a full tablet report...
W20260927 10:52:25.125878 20770 tablet.cc:2369] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:52:25.261823 21040 tablet.cc:2369] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:52:25.280652 20775 raft_consensus.cc:482] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:25.280961 20775 raft_consensus.cc:504] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.282177 20775 leader_election.cc:288] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
I20260927 10:52:25.286454 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.282503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:39607 (local address 127.6.31.131:45341)
0927 10:52:25.282657 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:25.282662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:25.282859 (+ 197us) server_negotiation.cc:407] Connection header received
0927 10:52:25.283031 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:25.283034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:25.283116 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:25.283198 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:25.283524 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.284116 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.285375 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.285829 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.285924 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.285965 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:25.286228 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 10:52:25.286314 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:52:25.286490 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.282405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:45341 (local address 127.6.31.129:39607)
0927 10:52:25.282757 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:25.282772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:25.282902 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:25.283221 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:25.283224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:25.283241 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:25.283409 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.283417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.284225 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.284229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.285249 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.285259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.285955 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.285959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.286064 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.286118 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 10:52:25.286199 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:52:25.286260 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":20,"thread_start_us":97,"threads_started":1}
I20260927 10:52:25.286530 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.282649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:47347 (local address 127.6.31.130:34137)
0927 10:52:25.282765 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:25.282771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:25.283034 (+ 263us) server_negotiation.cc:407] Connection header received
0927 10:52:25.283209 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:25.283212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:25.283266 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:25.283334 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:25.283622 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.284251 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.285490 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.285964 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.286075 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.286126 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:25.286335 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 10:52:25.286423 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:52:25.286509 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.282499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34137 (local address 127.6.31.129:47347)
0927 10:52:25.282911 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:25.282930 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:25.283046 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:25.283367 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:25.283371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:25.283392 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:25.283462 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.283487 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.284394 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.284399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.285368 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:25.285378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.286093 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.286096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:25.286196 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.286220 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:52:25.286296 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:52:25.286359 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260927 10:52:25.287281 20993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cdc5556e474b9296f482ce7bdbfff7" candidate_uuid: "42766c759ce44956a5ffcd71e43f36b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33307922e7416480614c958b045723" is_pre_election: true
I20260927 10:52:25.287451 20861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cdc5556e474b9296f482ce7bdbfff7" candidate_uuid: "42766c759ce44956a5ffcd71e43f36b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374a36d475454ef78c4ad803b22f0158" is_pre_election: true
I20260927 10:52:25.287645 20993 raft_consensus.cc:2454] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42766c759ce44956a5ffcd71e43f36b1 in term 0.
I20260927 10:52:25.287756 20861 raft_consensus.cc:2454] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42766c759ce44956a5ffcd71e43f36b1 in term 0.
I20260927 10:52:25.288136 20658 leader_election.cc:302] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42766c759ce44956a5ffcd71e43f36b1, db33307922e7416480614c958b045723; no voters:
I20260927 10:52:25.288463 20775 raft_consensus.cc:2781] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:52:25.288547 20775 raft_consensus.cc:482] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:52:25.288578 20775 raft_consensus.cc:3037] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:25.289623 20775 raft_consensus.cc:504] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.289976 20775 leader_election.cc:288] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 1 election: Requested vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
I20260927 10:52:25.290316 20861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cdc5556e474b9296f482ce7bdbfff7" candidate_uuid: "42766c759ce44956a5ffcd71e43f36b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374a36d475454ef78c4ad803b22f0158"
I20260927 10:52:25.290321 20993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cdc5556e474b9296f482ce7bdbfff7" candidate_uuid: "42766c759ce44956a5ffcd71e43f36b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33307922e7416480614c958b045723"
I20260927 10:52:25.290444 20993 raft_consensus.cc:3037] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:25.290444 20861 raft_consensus.cc:3037] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:25.291752 20993 raft_consensus.cc:2454] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42766c759ce44956a5ffcd71e43f36b1 in term 1.
I20260927 10:52:25.291752 20861 raft_consensus.cc:2454] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42766c759ce44956a5ffcd71e43f36b1 in term 1.
I20260927 10:52:25.292065 20657 leader_election.cc:302] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 374a36d475454ef78c4ad803b22f0158, 42766c759ce44956a5ffcd71e43f36b1; no voters:
I20260927 10:52:25.292265 20775 raft_consensus.cc:2781] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:52:25.292452 20775 raft_consensus.cc:686] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 LEADER]: Becoming Leader. State: Replica: 42766c759ce44956a5ffcd71e43f36b1, State: Running, Role: LEADER
I20260927 10:52:25.292714 20775 consensus_queue.cc:237] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.295822 20581 catalog_manager.cc:5495] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42766c759ce44956a5ffcd71e43f36b1 (127.6.31.129). New cstate: current_term: 1 leader_uuid: "42766c759ce44956a5ffcd71e43f36b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } health_report { overall_health: HEALTHY } } }
W20260927 10:52:25.317309 20908 tablet.cc:2369] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:52:25.353818 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.350068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.6.31.129:34075)
0927 10:52:25.350385 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:25.350391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:25.350404 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:52:25.350511 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:25.350515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:25.350591 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:25.350691 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:52:25.350908 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.351547 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.352327 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:25.352550 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:25.352630 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:25.352934 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:52:25.353386 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:52:25.353480 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:25.353545 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 10:52:25.353591 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
I20260927 10:52:25.358147 20861 raft_consensus.cc:1261] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [term 1 FOLLOWER]: Refusing update from remote peer 42766c759ce44956a5ffcd71e43f36b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:25.358178 20993 raft_consensus.cc:1261] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [term 1 FOLLOWER]: Refusing update from remote peer 42766c759ce44956a5ffcd71e43f36b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:25.358920 20775 consensus_queue.cc:1035] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:25.359123 21060 consensus_queue.cc:1035] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:25.368695 21066 mvcc.cc:205] Tried to move back new op lower bound from 7333913990579245056 to 7333913990321422336. Current Snapshot: MvccSnapshot[applied={T|T < 7333913990579245056}]
I20260927 10:52:25.374017 20580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:52:25.374197 20580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:52:25.376467 20580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Sending DeleteTablet for 3 replicas of tablet 51cdc5556e474b9296f482ce7bdbfff7
I20260927 10:52:25.377388 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 20619
I20260927 10:52:25.377568 20841 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cdc5556e474b9296f482ce7bdbfff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:25 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:38730
I20260927 10:52:25.378034 20973 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cdc5556e474b9296f482ce7bdbfff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:25 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:44868
I20260927 10:52:25.378526 21075 tablet_replica.cc:330] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: stopping tablet replica
I20260927 10:52:25.378789 21074 tablet_replica.cc:330] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: stopping tablet replica
I20260927 10:52:25.379024 21075 raft_consensus.cc:2229] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:52:25.379041 21074 raft_consensus.cc:2229] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:52:25.379393 21074 raft_consensus.cc:2258] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:52:25.379393 21075 raft_consensus.cc:2258] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:52:25.380769 21074 ts_tablet_manager.cc:1905] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:52:25.380849 21075 ts_tablet_manager.cc:1905] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:52:25.383023 21074 ts_tablet_manager.cc:1918] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:52:25.383168 21074 log.cc:1196] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/51cdc5556e474b9296f482ce7bdbfff7
I20260927 10:52:25.383466 21074 ts_tablet_manager.cc:1939] T 51cdc5556e474b9296f482ce7bdbfff7 P 374a36d475454ef78c4ad803b22f0158: Deleting consensus metadata
I20260927 10:52:25.384347 21075 ts_tablet_manager.cc:1918] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:52:25.384456 21075 log.cc:1196] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/51cdc5556e474b9296f482ce7bdbfff7
I20260927 10:52:25.384678 20567 catalog_manager.cc:4841] TS 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137): tablet 51cdc5556e474b9296f482ce7bdbfff7 (table test-table [id=1647c9c03a4c46b8a231516df7a6e35c]) successfully deleted
I20260927 10:52:25.384779 21075 ts_tablet_manager.cc:1939] T 51cdc5556e474b9296f482ce7bdbfff7 P db33307922e7416480614c958b045723: Deleting consensus metadata
W20260927 10:52:25.385092 20568 connection.cc:535] client connection to 127.6.31.129:34075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 10:52:25.385227 20568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:52:25.386276 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.385961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:34075 (local address 127.0.0.1:39262)
0927 10:52:25.386087 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:25.386180 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:34075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:52:25.386296 20568 catalog_manager.cc:4841] TS db33307922e7416480614c958b045723 (127.6.31.131:45341): tablet 51cdc5556e474b9296f482ce7bdbfff7 (table test-table [id=1647c9c03a4c46b8a231516df7a6e35c]) successfully deleted
I20260927 10:52:25.386332 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 20773
W20260927 10:52:25.386472 20568 catalog_manager.cc:4584] TS 42766c759ce44956a5ffcd71e43f36b1 (127.6.31.129:34075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51cdc5556e474b9296f482ce7bdbfff7: Network error: Client connection negotiation failed: client connection to 127.6.31.129:34075: connect: Connection refused (error 111)
I20260927 10:52:25.392077 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 20910
I20260927 10:52:25.397646 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 20548
I20260927 10:52:25.404271 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36581
--webserver_interface=127.6.31.190
--webserver_port=44665
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:36581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:25.514454 21077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:25.514789 21077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:25.514854 21077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:25.518915 21077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:25.519035 21077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:25.519074 21077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:25.519132 21077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:25.519191 21077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:25.524602 21077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:36581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=44665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:25.525719 21077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:25.526757 21077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:25.535844 21082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:25.535945 21085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:25.535967 21083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:25.536134 21077 server_base.cc:1034] running on GCE node
I20260927 10:52:25.536732 21077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:25.537879 21077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:25.539124 21077 hybrid_clock.cc:648] HybridClock initialized: now 1790506345539053 us; error 38 us; skew 500 ppm
Sep 27 10:52:25 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506345, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:25.542006 21077 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:52:25.542972 21077 webserver.cc:460] Webserver started at http://127.6.31.190:44665/ using document root <none> and password file <none>
I20260927 10:52:25.543579 21077 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:25.543674 21077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:25.547439 21077 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:52:25.549587 21092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:25.550962 21077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:52:25.551151 21077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "8976e07ff09c4e16b58dbc2b512a4504"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "333f1f36aed65d93211b3493c8ee588e"
server_key_iv: "05f066dae6ef62a9b10e9404593f9df9"
server_key_version: "encryptionkey@0"
I20260927 10:52:25.551715 21077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:25.566761 21077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:25.570297 21077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:25.570514 21077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:25.577868 21077 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:36581
I20260927 10:52:25.578148 21144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:36581 every 8 connection(s)
I20260927 10:52:25.579023 21077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:52:25.580513 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 21077
I20260927 10:52:25.580960 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:34075
--local_ip_for_outbound_sockets=127.6.31.129
--tserver_master_addrs=127.6.31.190:36581
--webserver_port=38395
--webserver_interface=127.6.31.129
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:52:25.584256 21145 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:52:25.586050 21145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Bootstrap starting.
I20260927 10:52:25.603490 21145 log.cc:824] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:25.614667 21145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:52:25.615070 21145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504: Bootstrap complete.
I20260927 10:52:25.619128 21145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:25.619567 21145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8976e07ff09c4e16b58dbc2b512a4504, State: Initialized, Role: FOLLOWER
I20260927 10:52:25.620096 21145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:25.620215 21145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:52:25.620297 21145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:52:25.620411 21145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:52:25.622282 21145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:25.622611 21145 leader_election.cc:302] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8976e07ff09c4e16b58dbc2b512a4504; no voters:
I20260927 10:52:25.622938 21145 leader_election.cc:288] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:52:25.623098 21149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:52:25.623345 21149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [term 3 LEADER]: Becoming Leader. State: Replica: 8976e07ff09c4e16b58dbc2b512a4504, State: Running, Role: LEADER
I20260927 10:52:25.623644 21149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } }
I20260927 10:52:25.624188 21145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:52:25.625167 21151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8976e07ff09c4e16b58dbc2b512a4504. Latest consensus state: current_term: 3 leader_uuid: "8976e07ff09c4e16b58dbc2b512a4504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } } }
I20260927 10:52:25.625331 21151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:25.625572 21150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8976e07ff09c4e16b58dbc2b512a4504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8976e07ff09c4e16b58dbc2b512a4504" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36581 } } }
I20260927 10:52:25.625646 21150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8976e07ff09c4e16b58dbc2b512a4504 [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:25.625689 21158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:52:25.630646 21158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:52:25.631327 21158 catalog_manager.cc:1259] Loaded cluster ID: 1ede30254f5d42a6987e58824736f3c1
I20260927 10:52:25.631386 21158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 10:52:25.632406 21158 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3924a679ad google::LogMessage::Fail() at ??:0
@ 0x7f3924a6a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3924a674de google::LogMessage::Flush() at ??:0
@ 0x7f3924a68a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f392f3a6fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f392f3a7d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f392f3a4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f392f3e8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3926cb93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3926cb9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3926cbb3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3926caa512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f39282ee6db start_thread at ??:0
@ 0x7f392356561f clone at ??:0
W20260927 10:52:25.711395 21147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:25.711752 21147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:25.711858 21147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:25.715926 21147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:25.716001 21147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:25.716096 21147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:52:25.721457 21147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:34075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=38395
--tserver_master_addrs=127.6.31.190:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:25.722527 21147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:25.723578 21147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:25.733368 21186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:25.733354 21187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:25.733565 21189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:25.733762 21147 server_base.cc:1034] running on GCE node
I20260927 10:52:25.734179 21147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:25.734869 21147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:25.736090 21147 hybrid_clock.cc:648] HybridClock initialized: now 1790506345736058 us; error 50 us; skew 500 ppm
Sep 27 10:52:25 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506345, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:25.739207 21147 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:52:25.740177 21147 webserver.cc:460] Webserver started at http://127.6.31.129:38395/ using document root <none> and password file <none>
I20260927 10:52:25.740828 21147 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:25.740926 21147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:25.745121 21147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:52:25.747357 21196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:25.748692 21147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:52:25.748871 21147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "42766c759ce44956a5ffcd71e43f36b1"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "34c10f8c1001f0b5860a264ae7b1865d"
server_key_iv: "e9b5546ecec134278e7251c2eb13f335"
server_key_version: "encryptionkey@0"
I20260927 10:52:25.749432 21147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:25.763664 21147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:25.767124 21147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:25.767347 21147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:25.768047 21147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:25.769376 21203 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:52:25.773728 21147 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:52:25.773825 21147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 10:52:25.773896 21147 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:52:25.776239 21147 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:52:25.776302 21147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 10:52:25.776707 21203 tablet_bootstrap.cc:492] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Bootstrap starting.
I20260927 10:52:25.787668 21147 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:34075
I20260927 10:52:25.787706 21310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:34075 every 8 connection(s)
I20260927 10:52:25.788883 21147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:52:25.796713 21203 log.cc:824] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:25.798615 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 21147
I20260927 10:52:25.799134 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:34137
--local_ip_for_outbound_sockets=127.6.31.130
--tserver_master_addrs=127.6.31.190:36581
--webserver_port=35975
--webserver_interface=127.6.31.130
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:52:25.804260 21203 tablet_bootstrap.cc:492] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:52:25.804634 21203 tablet_bootstrap.cc:492] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Bootstrap complete.
I20260927 10:52:25.805768 21203 ts_tablet_manager.cc:1397] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20260927 10:52:25.808163 21203 raft_consensus.cc:348] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.808589 21203 raft_consensus.cc:729] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42766c759ce44956a5ffcd71e43f36b1, State: Initialized, Role: FOLLOWER
I20260927 10:52:25.809075 21203 consensus_queue.cc:260] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:25.810005 21203 ts_tablet_manager.cc:1428] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260927 10:52:25.890641 21313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:52:25.791170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:43777)
0927 10:52:25.791768 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:25.791833 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:25.792970 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:25.889825 (+ 96855us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":144,"threads_started":1}
W20260927 10:52:25.891316 21311 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:36581 (0 consecutive failures): Network error: Failed to ping master at 127.6.31.190:36581: Client connection negotiation failed: client connection to 127.6.31.190:36581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:52:25.892091 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.891842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:39593)
0927 10:52:25.891971 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:25.892034 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 10:52:25.892776 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:25.892588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:50467)
0927 10:52:25.892687 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:25.892727 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:52:25.892980 21311 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:52:25.930097 21315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:25.930347 21315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:25.930421 21315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:25.934484 21315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:25.934542 21315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:25.934638 21315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:52:25.940064 21315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:34137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=35975
--tserver_master_addrs=127.6.31.190:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:25.941136 21315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:25.942181 21315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:25.951732 21324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:25.951738 21322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:25.951956 21315 server_base.cc:1034] running on GCE node
W20260927 10:52:25.951738 21321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:25.952503 21315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:25.953191 21315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:25.954361 21315 hybrid_clock.cc:648] HybridClock initialized: now 1790506345954341 us; error 31 us; skew 500 ppm
Sep 27 10:52:25 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506345, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:25.957127 21315 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:52:25.958068 21315 webserver.cc:460] Webserver started at http://127.6.31.130:35975/ using document root <none> and password file <none>
I20260927 10:52:25.958626 21315 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:25.958717 21315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:25.962451 21315 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:52:25.964576 21331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:25.965819 21315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:52:25.965947 21315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "374a36d475454ef78c4ad803b22f0158"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "2614df765363825ed093a1aa3cec6934"
server_key_iv: "6dad4089fb1fc9d248af6d62a80e0bd1"
server_key_version: "encryptionkey@0"
I20260927 10:52:25.966511 21315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:25.984462 21315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:25.987578 21315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:25.987797 21315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:25.988407 21315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:25.989238 21315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:25.989316 21315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:25.989389 21315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:25.989432 21315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:25.998235 21315 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:34137
I20260927 10:52:25.998286 21444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:34137 every 8 connection(s)
I20260927 10:52:25.999266 21315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:52:26.003055 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.001550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:42353)
0927 10:52:26.002222 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:26.002343 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":107,"threads_started":1}
W20260927 10:52:26.003680 21445 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:36581 (0 consecutive failures): Network error: Failed to ping master at 127.6.31.190:36581: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
I20260927 10:52:26.004297 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.004080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:32961)
0927 10:52:26.004186 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:26.004234 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 10:52:26.004902 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.004727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:59401)
0927 10:52:26.004817 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:26.004857 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 10:52:26.005098 21445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:52:26.008445 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 21315
I20260927 10:52:26.008836 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:45341
--local_ip_for_outbound_sockets=127.6.31.131
--tserver_master_addrs=127.6.31.190:36581
--webserver_port=45523
--webserver_interface=127.6.31.131
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:26.123826 21448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:26.124157 21448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:26.124239 21448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:26.128270 21448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:26.128331 21448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:26.128391 21448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:52:26.133576 21448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:45341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=45523
--tserver_master_addrs=127.6.31.190:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:26.134666 21448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:26.135663 21448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:52:26.145627 21448 server_base.cc:1034] running on GCE node
W20260927 10:52:26.145596 21453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:26.145588 21454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:26.145588 21456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:26.146238 21448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:26.146828 21448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:26.147990 21448 hybrid_clock.cc:648] HybridClock initialized: now 1790506346147969 us; error 32 us; skew 500 ppm
Sep 27 10:52:26 dist-test-slave-c60x krb5kdc[20000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506346, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:26.150884 21448 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:52:26.151926 21448 webserver.cc:460] Webserver started at http://127.6.31.131:45523/ using document root <none> and password file <none>
I20260927 10:52:26.152520 21448 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:26.152583 21448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:26.156785 21448 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:52:26.158828 21463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:26.160094 21448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:52:26.160234 21448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "db33307922e7416480614c958b045723"
format_stamp: "Formatted at 2026-09-27 10:52:23 on dist-test-slave-c60x"
server_key: "fbca3640156b8a97521ddbb5a7133808"
server_key_iv: "3a496d9e3cc99d419af1a36f09cb6ec0"
server_key_version: "encryptionkey@0"
I20260927 10:52:26.160807 21448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:26.172495 21448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:26.175688 21448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:26.175922 21448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:26.176539 21448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:26.177393 21448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:26.177465 21448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:26.177539 21448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:26.177583 21448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:26.187429 21448 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:45341
I20260927 10:52:26.187456 21576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:45341 every 8 connection(s)
I20260927 10:52:26.188696 21448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:52:26.193013 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.191308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:42229)
0927 10:52:26.192216 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:26.192325 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":138,"threads_started":1}
W20260927 10:52:26.193430 21577 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:36581 (0 consecutive failures): Network error: Failed to ping master at 127.6.31.190:36581: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
I20260927 10:52:26.194109 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.193883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:35765)
0927 10:52:26.193996 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:26.194047 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 10:52:26.194711 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.194514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:34679)
0927 10:52:26.194609 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:26.194653 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:52:26.194861 21577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:52:26.195492 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 21448
I20260927 10:52:26.894032 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.893584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:38387)
0927 10:52:26.893862 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:26.893919 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260927 10:52:26.975719 21582 raft_consensus.cc:482] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:26.975936 21582 raft_consensus.cc:504] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:26.977089 21582 leader_election.cc:288] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
Sep 27 10:52:26 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506345, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.131@KRBTEST.COM
Sep 27 10:52:26 dist-test-slave-c60x krb5kdc[20000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506345, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.130@KRBTEST.COM
I20260927 10:52:26.994266 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.977546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:57305 (local address 127.6.31.130:34137)
0927 10:52:26.977874 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:26.977886 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:26.977915 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:52:26.978777 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:26.978785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:26.979284 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:26.979612 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:26.981327 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:26.982893 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:26.984902 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:26.985605 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:26.985781 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:26.989607 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:26.989616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:26.989632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:26.989673 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:26.991739 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:26.992276 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:26.992283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:26.992289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:26.992436 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:26.992826 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:26.992829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:26.992830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:26.993193 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:26.993318 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:26.993825 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 10:52:26.994077 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260927 10:52:26.994257 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.977287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:45341 (local address 127.6.31.129:59123)
0927 10:52:26.977462 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:26.977476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:26.977582 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:26.979710 (+ 2128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:26.979717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:26.980343 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:26.981128 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:26.981147 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:26.983044 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:26.983047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:26.984764 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:26.984774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:26.985743 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:26.985747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:26.986176 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:26.987022 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:26.987049 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:26.988956 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:26.991538 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:26.991544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:26.991558 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:26.991915 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:26.992326 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:26.992332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:26.992336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:26.992495 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:26.993238 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:26.993246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:26.993515 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 10:52:26.993708 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 10:52:26.994062 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":604,"spinlock_wait_cycles":15488}
I20260927 10:52:26.994294 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.977430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:59123 (local address 127.6.31.131:45341)
0927 10:52:26.977842 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:26.977857 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:26.977906 (+ 49us) server_negotiation.cc:407] Connection header received
0927 10:52:26.978790 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:26.978797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:26.979284 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:26.979612 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:26.981327 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:26.982884 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:26.984902 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:26.985605 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:26.985781 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:26.989182 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:26.989189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:26.989210 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:26.989254 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:26.991423 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:26.992032 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:26.992042 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:26.992050 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:26.992201 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:26.992612 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:26.992615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:26.992617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:26.993117 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:26.993226 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:26.993824 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 10:52:26.994109 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":79,"threads_started":1}
I20260927 10:52:26.994319 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:26.977308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:34137 (local address 127.6.31.129:57305)
0927 10:52:26.977618 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:26.977635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:26.977734 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:26.979710 (+ 1976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:26.979717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:26.980343 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:26.981128 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:26.981147 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:26.983046 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:26.983048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:26.984763 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:26.984774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:26.985743 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:26.985747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:26.986176 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:26.987024 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:26.987051 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:26.989404 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:26.991871 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:26.991873 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:26.991874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:26.992144 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:26.992587 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:26.992593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:26.992595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:26.992699 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:26.993313 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:26.993315 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:26.993515 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 10:52:26.993709 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 10:52:26.994149 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":584,"spinlock_wait_cycles":69504,"thread_start_us":73,"threads_started":1}
W20260927 10:52:26.997187 21200 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db33307922e7416480614c958b045723 (127.6.31.131:45341): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
W20260927 10:52:26.997386 21199 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
I20260927 10:52:26.997484 21199 leader_election.cc:302] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42766c759ce44956a5ffcd71e43f36b1; no voters: 374a36d475454ef78c4ad803b22f0158, db33307922e7416480614c958b045723
I20260927 10:52:26.997771 21582 raft_consensus.cc:2726] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:52:27.006037 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:27.005625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:48401)
0927 10:52:27.005907 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:27.005956 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":122,"threads_started":1}
I20260927 10:52:27.196012 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:27.195517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:44531)
0927 10:52:27.195858 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:27.195915 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":91,"threads_started":1}
I20260927 10:52:27.895467 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:27.894913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:40745)
0927 10:52:27.895293 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:27.895368 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":86,"threads_started":1}
I20260927 10:52:28.007287 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:28.006785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:50991)
0927 10:52:28.007057 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:28.007181 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260927 10:52:28.197288 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:28.196865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:49067)
0927 10:52:28.197124 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:28.197179 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
I20260927 10:52:28.839917 21591 raft_consensus.cc:482] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:28.840047 21591 raft_consensus.cc:504] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:28.840606 21591 leader_election.cc:288] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
W20260927 10:52:28.841359 21200 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db33307922e7416480614c958b045723 (127.6.31.131:45341): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
W20260927 10:52:28.841459 21199 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
I20260927 10:52:28.841572 21199 leader_election.cc:302] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42766c759ce44956a5ffcd71e43f36b1; no voters: 374a36d475454ef78c4ad803b22f0158, db33307922e7416480614c958b045723
I20260927 10:52:28.841737 21591 raft_consensus.cc:2726] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:52:28.896697 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:28.896251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:57515)
0927 10:52:28.896532 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:28.896587 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260927 10:52:29.008358 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:29.007957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:33255)
0927 10:52:29.008194 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:29.008248 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260927 10:52:29.198493 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:29.198104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:44415)
0927 10:52:29.198350 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:29.198401 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260927 10:52:29.898003 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:29.897564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:49251)
0927 10:52:29.897824 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:29.897908 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260927 10:52:30.009752 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:30.009197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:33551)
0927 10:52:30.009541 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:30.009614 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":82,"threads_started":1}
I20260927 10:52:30.199792 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:30.199312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:35885)
0927 10:52:30.199577 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:30.199633 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260927 10:52:30.899402 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:30.898887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:59449)
0927 10:52:30.899222 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:30.899307 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":93,"threads_started":1}
I20260927 10:52:31.010921 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:31.010529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:42707)
0927 10:52:31.010767 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:31.010822 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260927 10:52:31.201046 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:31.200623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:60779)
0927 10:52:31.200887 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:31.200946 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260927 10:52:31.900599 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:31.900209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:39441)
0927 10:52:31.900452 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:31.900505 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260927 10:52:32.001663 21602 raft_consensus.cc:482] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:32.001788 21602 raft_consensus.cc:504] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:32.002249 21602 leader_election.cc:288] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
W20260927 10:52:32.002985 21199 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
W20260927 10:52:32.003127 21200 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db33307922e7416480614c958b045723 (127.6.31.131:45341): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
I20260927 10:52:32.003225 21200 leader_election.cc:302] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42766c759ce44956a5ffcd71e43f36b1; no voters: 374a36d475454ef78c4ad803b22f0158, db33307922e7416480614c958b045723
I20260927 10:52:32.003384 21602 raft_consensus.cc:2726] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:52:32.012120 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:32.011656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:54253)
0927 10:52:32.011969 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:32.012022 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":153,"threads_started":1}
I20260927 10:52:32.202419 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:32.201978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:50307)
0927 10:52:32.202260 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:32.202320 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
I20260927 10:52:32.901942 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:32.901523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:49059)
0927 10:52:32.901776 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:32.901836 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260927 10:52:33.013438 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:33.012894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:46975)
0927 10:52:33.013282 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:33.013341 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":107,"threads_started":1}
I20260927 10:52:33.203720 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:33.203252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:39817)
0927 10:52:33.203557 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:33.203624 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
I20260927 10:52:33.903451 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:33.902896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:55049)
0927 10:52:33.903259 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:33.903343 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":97,"threads_started":1}
I20260927 10:52:34.014937 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:34.014356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:38935)
0927 10:52:34.014726 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:34.014805 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":92,"threads_started":1}
I20260927 10:52:34.204926 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:34.204423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:36813)
0927 10:52:34.204763 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:34.204824 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260927 10:52:34.425633 21611 raft_consensus.cc:482] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:34.425770 21611 raft_consensus.cc:504] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:34.426510 21611 leader_election.cc:288] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
W20260927 10:52:34.427454 21200 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db33307922e7416480614c958b045723 (127.6.31.131:45341): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
W20260927 10:52:34.427618 21199 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
I20260927 10:52:34.427690 21199 leader_election.cc:302] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42766c759ce44956a5ffcd71e43f36b1; no voters: 374a36d475454ef78c4ad803b22f0158, db33307922e7416480614c958b045723
I20260927 10:52:34.427886 21611 raft_consensus.cc:2726] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:52:34.904927 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:34.904422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:38849)
0927 10:52:34.904767 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:34.904830 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
I20260927 10:52:35.016294 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:35.015808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:34009)
0927 10:52:35.016130 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:35.016194 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":85,"threads_started":1}
I20260927 10:52:35.206245 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:35.205820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:46419)
0927 10:52:35.206089 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:35.206148 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260927 10:52:35.906433 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:35.905977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:47245)
0927 10:52:35.906275 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:35.906337 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260927 10:52:36.017550 21616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:36.017101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:50377)
0927 10:52:36.017375 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:36.017451 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260927 10:52:36.207593 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:36.207167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:53309)
0927 10:52:36.207443 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:36.207494 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":109,"threads_started":1}
I20260927 10:52:36.907837 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:36.907379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:39551)
0927 10:52:36.907673 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:36.907737 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260927 10:52:37.018813 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:37.018379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:55869)
0927 10:52:37.018652 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:37.018713 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260927 10:52:37.209018 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:37.208536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:36933)
0927 10:52:37.208837 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:37.208909 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260927 10:52:37.908985 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:37.908561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:36547)
0927 10:52:37.908838 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:37.908892 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
I20260927 10:52:38.020237 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:38.019597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:60089)
0927 10:52:38.020006 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:38.020077 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":167,"threads_started":1}
I20260927 10:52:38.210284 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:38.209849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:52429)
0927 10:52:38.210127 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:38.210185 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
I20260927 10:52:38.542263 21624 raft_consensus.cc:482] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:38.542402 21624 raft_consensus.cc:504] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:38.542817 21624 leader_election.cc:288] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
W20260927 10:52:38.543669 21200 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db33307922e7416480614c958b045723 (127.6.31.131:45341): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
W20260927 10:52:38.543781 21199 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
I20260927 10:52:38.543897 21199 leader_election.cc:302] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42766c759ce44956a5ffcd71e43f36b1; no voters: 374a36d475454ef78c4ad803b22f0158, db33307922e7416480614c958b045723
I20260927 10:52:38.544065 21624 raft_consensus.cc:2726] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:52:38.910310 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:38.909866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:60837)
0927 10:52:38.910153 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:38.910211 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260927 10:52:39.021582 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:39.021102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:48105)
0927 10:52:39.021423 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:39.021485 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260927 10:52:39.211601 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:39.211054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:46265)
0927 10:52:39.211439 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:39.211497 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260927 10:52:39.911572 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:39.911160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:36423)
0927 10:52:39.911409 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:39.911463 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260927 10:52:40.022938 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:40.022384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:56723)
0927 10:52:40.022721 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:40.022798 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260927 10:52:40.212836 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:40.212413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.131:34843)
0927 10:52:40.212666 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:40.212726 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260927 10:52:40.493894 21631 raft_consensus.cc:482] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:40.494091 21631 raft_consensus.cc:504] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db33307922e7416480614c958b045723" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 45341 } } peers { permanent_uuid: "374a36d475454ef78c4ad803b22f0158" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 34137 } } peers { permanent_uuid: "42766c759ce44956a5ffcd71e43f36b1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 34075 } }
I20260927 10:52:40.494526 21631 leader_election.cc:288] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db33307922e7416480614c958b045723 (127.6.31.131:45341), 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137)
W20260927 10:52:40.495419 21200 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db33307922e7416480614c958b045723 (127.6.31.131:45341): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
W20260927 10:52:40.495522 21199 leader_election.cc:341] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374a36d475454ef78c4ad803b22f0158 (127.6.31.130:34137): Not found: Tablet not found: 51cdc5556e474b9296f482ce7bdbfff7
I20260927 10:52:40.495590 21199 leader_election.cc:302] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42766c759ce44956a5ffcd71e43f36b1; no voters: 374a36d475454ef78c4ad803b22f0158, db33307922e7416480614c958b045723
I20260927 10:52:40.495720 21631 raft_consensus.cc:2726] T 51cdc5556e474b9296f482ce7bdbfff7 P 42766c759ce44956a5ffcd71e43f36b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:52:40.913013 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:40.912582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.129:39997)
0927 10:52:40.912859 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:40.912918 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
I20260927 10:52:41.024322 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:41.023843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:36581 (local address 127.6.31.130:60195)
0927 10:52:41.024178 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:41.024226 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":87,"threads_started":1}
I20260927 10:52:41.199482 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 21147
I20260927 10:52:41.205468 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 21315
I20260927 10:52:41.211061 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 21448
2026-09-27T10:52:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19270 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:52:41 dist-test-slave-c60x krb5kdc[21636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:52:41 dist-test-slave-c60x krb5kdc[21636](info): set up 2 sockets
Sep 27 10:52:41 dist-test-slave-c60x krb5kdc[21636](info): commencing operation
krb5kdc: starting...
W20260927 10:52:42.255370 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:52:42 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:52:42.383584 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35191
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:42.493854 21649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:42.494167 21649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:42.494242 21649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:42.498421 21649 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:52:42.498517 21649 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:52:42.498543 21649 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:52:42.498569 21649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:52:42.498611 21649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:42.498656 21649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:42.498693 21649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:52:42.498728 21649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:52:42.504217 21649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:42.505427 21649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:42.506474 21649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:42.515816 21657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:42.515816 21649 server_base.cc:1034] running on GCE node
W20260927 10:52:42.515762 21655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:42.515851 21654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:42.516703 21649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:42.517894 21649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:42.519120 21649 hybrid_clock.cc:648] HybridClock initialized: now 1790506362519064 us; error 32 us; skew 500 ppm
Sep 27 10:52:42 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:42.530316 21649 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:52:42.531383 21649 webserver.cc:460] Webserver started at http://127.6.31.190:43041/ using document root <none> and password file <none>
I20260927 10:52:42.531976 21649 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:42.532071 21649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:42.532308 21649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:42.534509 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "72c8ad9989db4b69b587ed6fdca6794d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "bb71cfb47a8220966a8f9805b5317b57"
server_key_iv: "57c96abbe08a365e336a523509b7a604"
server_key_version: "encryptionkey@0"
I20260927 10:52:42.535153 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "72c8ad9989db4b69b587ed6fdca6794d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "bb71cfb47a8220966a8f9805b5317b57"
server_key_iv: "57c96abbe08a365e336a523509b7a604"
server_key_version: "encryptionkey@0"
I20260927 10:52:42.538949 21649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:52:42.541554 21664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:42.542825 21649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:52:42.543004 21649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "72c8ad9989db4b69b587ed6fdca6794d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "bb71cfb47a8220966a8f9805b5317b57"
server_key_iv: "57c96abbe08a365e336a523509b7a604"
server_key_version: "encryptionkey@0"
I20260927 10:52:42.543169 21649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:42.562853 21649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:42.566100 21649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:42.566349 21649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:42.573324 21649 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:35191
I20260927 10:52:42.573607 21716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:35191 every 8 connection(s)
I20260927 10:52:42.574369 21649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:52:42.577296 21717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:42.580302 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 21649
I20260927 10:52:42.580439 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:52:42.580725 6270 external_mini_cluster.cc:1428] Setting key 915be59e50a80abc40a5b22f9f1b517d
I20260927 10:52:42.583428 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Bootstrap starting.
I20260927 10:52:42.586130 21717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:42.586946 21717 log.cc:824] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:42.589591 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: No bootstrap required, opened a new log
Sep 27 10:52:42 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:42.593601 21717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:52:42.593921 21717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:42.593971 21717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c8ad9989db4b69b587ed6fdca6794d, State: Initialized, Role: FOLLOWER
I20260927 10:52:42.594559 21717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:52:42.594739 21717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:52:42.594810 21717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:52:42.594921 21717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:42.596230 21717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:52:42.596611 21717 leader_election.cc:302] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72c8ad9989db4b69b587ed6fdca6794d; no voters:
I20260927 10:52:42.596998 21717 leader_election.cc:288] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:52:42.597167 21722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:52:42.597391 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:42.582192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.6.31.190:35191)
0927 10:52:42.582610 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:42.582633 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:42.582674 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:52:42.583517 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:42.583539 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:42.583963 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:42.584303 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:42.584985 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:42.586441 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:42.587599 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:42.588379 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:42.588534 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:42.591241 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:42.591248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:42.591266 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:42.591299 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:42.593951 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:42.594467 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:42.594473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:42.594479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:42.594624 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:42.594984 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:42.594988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:42.594990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:42.595564 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:42.595777 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:42.595996 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 10:52:42.596339 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260927 10:52:42.597455 21722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 1 LEADER]: Becoming Leader. State: Replica: 72c8ad9989db4b69b587ed6fdca6794d, State: Running, Role: LEADER
I20260927 10:52:42.597998 21722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:52:42.598299 21717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:52:42.599885 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72c8ad9989db4b69b587ed6fdca6794d. Latest consensus state: current_term: 1 leader_uuid: "72c8ad9989db4b69b587ed6fdca6794d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } } }
I20260927 10:52:42.600121 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:42.599928 21723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72c8ad9989db4b69b587ed6fdca6794d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } } }
I20260927 10:52:42.600489 21723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: This master's current role is: LEADER
I20260927 10:52:42.600555 21736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:52:42.603777 21736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:52:42.609692 21736 catalog_manager.cc:1347] Generated new cluster ID: d9b8e1780aef4094a9e983d69e974905
I20260927 10:52:42.609782 21736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:52:42.627357 21736 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:52:42.628389 21736 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:52:42.633229 21736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Generated new TSK 0
I20260927 10:52:42.634071 21736 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:52:42.670579 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:35191
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:42.783401 21743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:42.783664 21743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:42.783707 21743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:42.787784 21743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:42.787840 21743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:42.787937 21743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:52:42.793231 21743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:42.794292 21743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:42.795369 21743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:42.805287 21748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:42.805300 21749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:42.805576 21743 server_base.cc:1034] running on GCE node
W20260927 10:52:42.805740 21751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:42.806171 21743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:42.806802 21743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:42.807986 21743 hybrid_clock.cc:648] HybridClock initialized: now 1790506362807955 us; error 41 us; skew 500 ppm
Sep 27 10:52:42 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:42.810616 21743 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:52:42.811525 21743 webserver.cc:460] Webserver started at http://127.6.31.129:35433/ using document root <none> and password file <none>
I20260927 10:52:42.812076 21743 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:42.812165 21743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:42.812399 21743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:42.814405 21743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "b396f34c2db746d4b50c39834982262d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "c3e2279462b5f5e75b93e6ac9c9ea936"
server_key_iv: "21b9a747d43dbb386cb08029eb356138"
server_key_version: "encryptionkey@0"
I20260927 10:52:42.814980 21743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "b396f34c2db746d4b50c39834982262d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "c3e2279462b5f5e75b93e6ac9c9ea936"
server_key_iv: "21b9a747d43dbb386cb08029eb356138"
server_key_version: "encryptionkey@0"
I20260927 10:52:42.818795 21743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 10:52:42.821259 21758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:42.822443 21743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:52:42.822578 21743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "b396f34c2db746d4b50c39834982262d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "c3e2279462b5f5e75b93e6ac9c9ea936"
server_key_iv: "21b9a747d43dbb386cb08029eb356138"
server_key_version: "encryptionkey@0"
I20260927 10:52:42.822697 21743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:42.835026 21743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:42.838506 21743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:42.838728 21743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:42.839437 21743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:42.840268 21743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:42.840348 21743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:42.840420 21743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:42.840610 21743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:42.849942 21743 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:40935
I20260927 10:52:42.849961 21871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:40935 every 8 connection(s)
I20260927 10:52:42.850867 21743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:52:42.857343 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 21743
I20260927 10:52:42.857455 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:52:42.857728 6270 external_mini_cluster.cc:1428] Setting key e9c80dbe489fdfcd71b9cc86b6b4831c
Sep 27 10:52:42 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:42.865880 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:42.852864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:36975 (local address 127.6.31.190:35191)
0927 10:52:42.853010 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:42.853013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:42.853807 (+ 794us) server_negotiation.cc:407] Connection header received
0927 10:52:42.854986 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:42.854989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:42.855036 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:42.855194 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:42.856359 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:42.856999 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:42.858371 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:42.858647 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:42.858738 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:42.861661 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:42.861666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:42.861669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:42.861705 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:42.863559 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:42.864118 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:42.864125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:42.864129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:42.864233 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:42.864596 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:42.864602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:42.864605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:42.864896 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:42.865027 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:42.865538 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 10:52:42.865690 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:52:42.866664 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:42.853174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:36975)
0927 10:52:42.853621 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:42.853668 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:42.854792 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:42.855342 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:42.855349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:42.855731 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:42.856215 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:42.856228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:42.857129 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:42.857131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:42.858233 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:42.858248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:42.858803 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:42.858807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:42.859007 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:42.859752 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:42.859770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:42.861478 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:42.863698 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:42.863704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:42.863718 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:42.864005 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:42.864347 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:42.864349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:42.864350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:42.864495 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:42.865021 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:42.865026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:42.865288 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 10:52:42.865451 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 10:52:42.865735 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:52:42.868001 21872 heartbeater.cc:347] Connected to a master server at 127.6.31.190:35191
I20260927 10:52:42.868281 21872 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:42.868858 21872 heartbeater.cc:511] Master 127.6.31.190:35191 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:52:42.870594 21681 ts_manager.cc:194] Registered new tserver with Master: b396f34c2db746d4b50c39834982262d (127.6.31.129:40935)
I20260927 10:52:42.872030 21681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:36975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:52:42.890528 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:35191
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:43.005717 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:43.006037 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:43.006129 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:43.010227 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:43.010346 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:43.010448 21877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:52:43.016026 21877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:43.017161 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:43.018213 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:43.028120 21885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:43.028151 21882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:43.028225 21877 server_base.cc:1034] running on GCE node
W20260927 10:52:43.028447 21883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:43.028806 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:43.029446 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:43.030642 21877 hybrid_clock.cc:648] HybridClock initialized: now 1790506363030618 us; error 39 us; skew 500 ppm
Sep 27 10:52:43 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506363, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:43.033556 21877 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:52:43.034564 21877 webserver.cc:460] Webserver started at http://127.6.31.130:32925/ using document root <none> and password file <none>
I20260927 10:52:43.035194 21877 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:43.035286 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:43.035512 21877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:43.037683 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "2edcf0fb871243b5a5fc3c0816c3040d"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "1f0c1757baa66a47c6399d21d3be8a8b"
server_key_iv: "fad6dc80056b887d71346d4561b34250"
server_key_version: "encryptionkey@0"
I20260927 10:52:43.038266 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "2edcf0fb871243b5a5fc3c0816c3040d"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "1f0c1757baa66a47c6399d21d3be8a8b"
server_key_iv: "fad6dc80056b887d71346d4561b34250"
server_key_version: "encryptionkey@0"
I20260927 10:52:43.042183 21877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 10:52:43.044891 21892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:43.046224 21877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:52:43.046368 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "2edcf0fb871243b5a5fc3c0816c3040d"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "1f0c1757baa66a47c6399d21d3be8a8b"
server_key_iv: "fad6dc80056b887d71346d4561b34250"
server_key_version: "encryptionkey@0"
I20260927 10:52:43.046538 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:43.062024 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:43.065630 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:43.065858 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:43.066461 21877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:43.067399 21877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:43.067469 21877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:43.067534 21877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:43.067575 21877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:43.077014 21877 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:33687
I20260927 10:52:43.077051 22005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:33687 every 8 connection(s)
I20260927 10:52:43.077934 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:52:43.087390 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 21877
I20260927 10:52:43.087512 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:52:43.087829 6270 external_mini_cluster.cc:1428] Setting key 35263d7d908c406dec13b70bf994a0a1
Sep 27 10:52:43 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506363, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:43.092777 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.079841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:49743 (local address 127.6.31.190:35191)
0927 10:52:43.080015 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:43.080020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:43.080946 (+ 926us) server_negotiation.cc:407] Connection header received
0927 10:52:43.082207 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:43.082210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:43.082259 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:43.082341 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:43.083587 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.084151 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.085282 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.085513 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.085599 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.088571 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:43.088575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:43.088577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:43.088604 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:43.090345 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:43.090916 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:43.090922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:43.090927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:43.091025 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:43.091536 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:43.091540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:43.091542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:43.091746 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:43.091866 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:43.092440 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 10:52:43.092595 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 10:52:43.093601 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.080252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:49743)
0927 10:52:43.080788 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:43.080834 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:43.081989 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:43.082531 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:43.082538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:43.082912 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:43.083420 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.083435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.084273 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.084275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.085158 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.085167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.085654 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.085657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.085799 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.086452 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:43.086469 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:43.088380 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:43.090488 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:43.090493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:43.090507 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:43.090806 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:43.091199 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:43.091203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:43.091204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:43.091419 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:43.091869 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:43.091879 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:43.092149 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 10:52:43.092365 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 10:52:43.092725 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":102,"threads_started":1}
I20260927 10:52:43.094873 22006 heartbeater.cc:347] Connected to a master server at 127.6.31.190:35191
I20260927 10:52:43.095192 22006 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:43.095865 22006 heartbeater.cc:511] Master 127.6.31.190:35191 requested a full tablet report, sending...
I20260927 10:52:43.097162 21681 ts_manager.cc:194] Registered new tserver with Master: 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687)
I20260927 10:52:43.098083 21681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:49743
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:52:43.118808 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:35191
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:52:43.229533 22011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:52:43.229835 22011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:52:43.229911 22011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:52:43.234166 22011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:52:43.234246 22011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:52:43.234351 22011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:52:43.239923 22011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:52:43.241127 22011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:52:43.242182 22011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:52:43.252019 22016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:52:43.251991 22017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:43.252044 22011 server_base.cc:1034] running on GCE node
W20260927 10:52:43.251983 22019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:52:43.252665 22011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:52:43.253229 22011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:52:43.254420 22011 hybrid_clock.cc:648] HybridClock initialized: now 1790506363254409 us; error 35 us; skew 500 ppm
Sep 27 10:52:43 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506363, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:52:43.257409 22011 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:52:43.258394 22011 webserver.cc:460] Webserver started at http://127.6.31.131:38443/ using document root <none> and password file <none>
I20260927 10:52:43.258929 22011 fs_manager.cc:359] Metadata directory not provided
I20260927 10:52:43.258986 22011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:52:43.259213 22011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:52:43.261196 22011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "9d60676740dc4804b2882521469db5c5"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "ccbbac155d94e55cf458ef1786ad7677"
server_key_iv: "74838119368e9cd3a40d8a5c44a03348"
server_key_version: "encryptionkey@0"
I20260927 10:52:43.261749 22011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "9d60676740dc4804b2882521469db5c5"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "ccbbac155d94e55cf458ef1786ad7677"
server_key_iv: "74838119368e9cd3a40d8a5c44a03348"
server_key_version: "encryptionkey@0"
I20260927 10:52:43.265452 22011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:52:43.267795 22026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:43.269065 22011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:52:43.269207 22011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "9d60676740dc4804b2882521469db5c5"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "ccbbac155d94e55cf458ef1786ad7677"
server_key_iv: "74838119368e9cd3a40d8a5c44a03348"
server_key_version: "encryptionkey@0"
I20260927 10:52:43.269335 22011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:52:43.286473 22011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:52:43.289520 22011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:52:43.289733 22011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:52:43.290344 22011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:52:43.291281 22011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:52:43.291352 22011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:43.291424 22011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:52:43.291476 22011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:52:43.300719 22011 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:38909
I20260927 10:52:43.300740 22139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:38909 every 8 connection(s)
I20260927 10:52:43.301633 22011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:52:43.305289 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 22011
I20260927 10:52:43.305434 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:52:43.305724 6270 external_mini_cluster.cc:1428] Setting key e691863f77becf76de72c53dac875c5d
Sep 27 10:52:43 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506363, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:52:43.315609 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.303472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:42451 (local address 127.6.31.190:35191)
0927 10:52:43.303613 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:43.303617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:43.304467 (+ 850us) server_negotiation.cc:407] Connection header received
0927 10:52:43.305822 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:43.305825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:43.305883 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:43.305964 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:43.307443 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.308063 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.309178 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.309410 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.309530 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.312112 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:43.312116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:43.312118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:43.312144 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:43.313501 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:43.314002 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:43.314006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:43.314008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:43.314094 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:43.314427 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:43.314449 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:43.314451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:43.314634 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:43.314749 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:43.315302 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 10:52:43.315427 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:52:43.316462 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.303853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:42451)
0927 10:52:43.304303 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:43.304345 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:43.305616 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:43.306181 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:43.306191 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:43.306703 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:52:43.307295 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.307310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.308189 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.308197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.309052 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.309061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.309563 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.309567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.309739 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.310347 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:52:43.310364 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:52:43.311918 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:52:43.313621 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:43.313627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:43.313641 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:43.313894 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:43.314186 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:52:43.314191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:52:43.314193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:52:43.314331 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:52:43.314746 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:52:43.314752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:52:43.315006 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 10:52:43.315262 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 10:52:43.315580 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":103,"threads_started":1}
I20260927 10:52:43.317657 22140 heartbeater.cc:347] Connected to a master server at 127.6.31.190:35191
I20260927 10:52:43.317898 22140 heartbeater.cc:464] Registering TS with master...
I20260927 10:52:43.318454 22140 heartbeater.cc:511] Master 127.6.31.190:35191 requested a full tablet report, sending...
I20260927 10:52:43.319645 21680 ts_manager.cc:194] Registered new tserver with Master: 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909)
I20260927 10:52:43.320328 21680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:42451
I20260927 10:52:43.332156 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:52:43.342233 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.334315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.6.31.190:35191)
0927 10:52:43.334477 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:43.334483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:43.334674 (+ 191us) server_negotiation.cc:407] Connection header received
0927 10:52:43.334755 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:43.334758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:43.334803 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:43.334877 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:52:43.335657 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.336319 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.337218 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.337437 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.337517 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.338877 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:52:43.338880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:52:43.338882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:52:43.338905 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:52:43.340403 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:43.340879 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:43.340882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:43.340883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:43.340956 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:52:43.341309 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:52:43.341312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:52:43.341313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:52:43.341568 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:52:43.341690 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:43.341895 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 10:52:43.342006 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:52:43.345629 21680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:52:43.348969 21680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:52:43.363675 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.358123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:33687 (local address 127.0.0.1:37416)
0927 10:52:43.358951 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:43.358965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:43.359139 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:43.359623 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:43.359625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:43.359644 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:43.359744 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.359750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.361010 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.361017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.362067 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.362077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.363270 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.363274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.363387 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.363407 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:52:43.363473 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:52:43.363522 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260927 10:52:43.363806 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.358353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.6.31.130:33687)
0927 10:52:43.358779 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:43.358791 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:43.359049 (+ 258us) server_negotiation.cc:407] Connection header received
0927 10:52:43.359278 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:43.359282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:43.359500 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:43.359646 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:43.359847 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.360888 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.362196 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.363071 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.363204 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.363334 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:43.363519 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 10:52:43.363645 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":126,"threads_started":1}
I20260927 10:52:43.363999 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.358058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.6.31.131:38909)
0927 10:52:43.358988 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:43.359002 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:43.359023 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:52:43.359127 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:43.359132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:43.359295 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:43.359434 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:43.359669 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.360470 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.362316 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.362896 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.363019 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.363191 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:43.363740 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 10:52:43.363860 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":163,"threads_started":1}
I20260927 10:52:43.364032 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.357600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.6.31.129:40935)
0927 10:52:43.358090 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:43.358105 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:43.358469 (+ 364us) server_negotiation.cc:407] Connection header received
0927 10:52:43.358554 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:43.358560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:43.358768 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:43.358921 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:43.359432 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.360640 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.362294 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.363071 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.363204 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.363334 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:43.363771 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 10:52:43.363874 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":86,"threads_started":1}
I20260927 10:52:43.363674 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.357773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38909 (local address 127.0.0.1:44964)
0927 10:52:43.358416 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:43.358432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:43.358524 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:43.359444 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:43.359447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:43.359462 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:43.359546 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.359555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.360605 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.360608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.362171 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.362182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.363027 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.363031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.363191 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.363258 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 10:52:43.363372 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 10:52:43.363502 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":81,"thread_start_us":107,"threads_started":1}
I20260927 10:52:43.364317 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.357407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:40935 (local address 127.0.0.1:47586)
0927 10:52:43.358125 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:43.358177 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:43.358324 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:43.359131 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:43.359136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:43.359158 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:43.359300 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.359309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.360769 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.360772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.362171 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.362182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.363406 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.363410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.363534 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.363559 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:52:43.363632 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:52:43.363697 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":65,"thread_start_us":87,"threads_started":1}
I20260927 10:52:43.367265 21806 tablet_service.cc:1467] Processing CreateTablet for tablet c66a2085e8d24137af52f9ac724f74a0 (DEFAULT_TABLE table=test-table [id=7dd200016c27408f96461f82609ab73c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:43.367290 22074 tablet_service.cc:1467] Processing CreateTablet for tablet c66a2085e8d24137af52f9ac724f74a0 (DEFAULT_TABLE table=test-table [id=7dd200016c27408f96461f82609ab73c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:43.367290 21940 tablet_service.cc:1467] Processing CreateTablet for tablet c66a2085e8d24137af52f9ac724f74a0 (DEFAULT_TABLE table=test-table [id=7dd200016c27408f96461f82609ab73c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:52:43.368327 21806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66a2085e8d24137af52f9ac724f74a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:43.368367 22074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66a2085e8d24137af52f9ac724f74a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:43.368359 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66a2085e8d24137af52f9ac724f74a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:52:43.374886 22157 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Bootstrap starting.
I20260927 10:52:43.376075 22158 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: Bootstrap starting.
I20260927 10:52:43.377301 22157 tablet_bootstrap.cc:654] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:43.378053 22157 log.cc:824] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:43.378266 22158 tablet_bootstrap.cc:654] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:43.378756 22159 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Bootstrap starting.
I20260927 10:52:43.379107 22158 log.cc:824] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:43.380342 22157 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: No bootstrap required, opened a new log
I20260927 10:52:43.380563 22157 ts_tablet_manager.cc:1397] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:52:43.381039 22158 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: No bootstrap required, opened a new log
I20260927 10:52:43.381165 22159 tablet_bootstrap.cc:654] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Neither blocks nor log segments found. Creating new log.
I20260927 10:52:43.381214 22158 ts_tablet_manager.cc:1397] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:52:43.381846 22159 log.cc:824] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:52:43.383515 22159 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: No bootstrap required, opened a new log
I20260927 10:52:43.383713 22159 ts_tablet_manager.cc:1397] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:52:43.385178 22157 raft_consensus.cc:348] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:52:43.385178 22158 raft_consensus.cc:348] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:52:43.385474 22157 raft_consensus.cc:374] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:43.385474 22158 raft_consensus.cc:374] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:43.385535 22158 raft_consensus.cc:729] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d60676740dc4804b2882521469db5c5, State: Initialized, Role: FOLLOWER
I20260927 10:52:43.385576 22157 raft_consensus.cc:729] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b396f34c2db746d4b50c39834982262d, State: Initialized, Role: FOLLOWER
I20260927 10:52:43.386040 22157 consensus_queue.cc:260] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:52:43.386040 22158 consensus_queue.cc:260] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:52:43.387058 22158 ts_tablet_manager.cc:1428] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:52:43.387058 22157 ts_tablet_manager.cc:1428] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:52:43.386935 22159 raft_consensus.cc:348] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:52:43.387436 22140 heartbeater.cc:503] Master 127.6.31.190:35191 was elected leader, sending a full tablet report...
I20260927 10:52:43.387468 21872 heartbeater.cc:503] Master 127.6.31.190:35191 was elected leader, sending a full tablet report...
I20260927 10:52:43.388509 22159 raft_consensus.cc:374] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:52:43.388664 22159 raft_consensus.cc:729] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edcf0fb871243b5a5fc3c0816c3040d, State: Initialized, Role: FOLLOWER
I20260927 10:52:43.389106 22159 consensus_queue.cc:260] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:52:43.389778 22006 heartbeater.cc:503] Master 127.6.31.190:35191 was elected leader, sending a full tablet report...
I20260927 10:52:43.390280 22159 ts_tablet_manager.cc:1428] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 10:52:43.552610 22141 tablet.cc:2369] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:52:43.579291 22007 tablet.cc:2369] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:52:43.590516 22165 raft_consensus.cc:482] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:52:43.590719 22165 raft_consensus.cc:504] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:52:43.591969 22165 leader_election.cc:288] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b396f34c2db746d4b50c39834982262d (127.6.31.129:40935), 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909)
I20260927 10:52:43.595976 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.592498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:47057 (local address 127.6.31.131:38909)
0927 10:52:43.592614 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:43.592619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:43.592633 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:52:43.592693 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:43.592699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:43.592757 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:43.592844 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:43.593149 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.593710 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.594905 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.595416 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.595506 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.595586 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:43.595775 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 10:52:43.595854 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:52:43.595990 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.592209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38909 (local address 127.6.31.130:47057)
0927 10:52:43.592364 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:43.592380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:43.592474 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:43.592861 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:43.592864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:43.592884 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:43.593033 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.593040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.593817 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.593820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.594675 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.594685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.595531 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.595538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.595639 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.595668 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:52:43.595738 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:52:43.595820 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260927 10:52:43.596017 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.592242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:40935 (local address 127.6.31.130:35601)
0927 10:52:43.592650 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:43.592663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:52:43.592765 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:52:43.593047 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:52:43.593050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:52:43.593065 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:52:43.593186 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.593192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.593946 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.593948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.594716 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:52:43.594724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.595595 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.595600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:52:43.595700 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.595738 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:52:43.595817 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 10:52:43.595889 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":107,"thread_start_us":120,"threads_started":1}
I20260927 10:52:43.596030 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.592415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:35601 (local address 127.6.31.129:40935)
0927 10:52:43.592666 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:43.592670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:43.592724 (+ 54us) server_negotiation.cc:407] Connection header received
0927 10:52:43.592882 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:43.592884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:43.592934 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:43.593016 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:52:43.593299 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.593824 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.594967 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.595443 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.595583 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.595636 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:43.595851 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 10:52:43.595921 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260927 10:52:43.596750 22094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66a2085e8d24137af52f9ac724f74a0" candidate_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d60676740dc4804b2882521469db5c5" is_pre_election: true
I20260927 10:52:43.596758 21826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66a2085e8d24137af52f9ac724f74a0" candidate_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b396f34c2db746d4b50c39834982262d" is_pre_election: true
I20260927 10:52:43.597050 22094 raft_consensus.cc:2454] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edcf0fb871243b5a5fc3c0816c3040d in term 0.
I20260927 10:52:43.597076 21826 raft_consensus.cc:2454] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edcf0fb871243b5a5fc3c0816c3040d in term 0.
I20260927 10:52:43.597580 21896 leader_election.cc:302] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2edcf0fb871243b5a5fc3c0816c3040d, 9d60676740dc4804b2882521469db5c5, b396f34c2db746d4b50c39834982262d; no voters:
I20260927 10:52:43.597867 22165 raft_consensus.cc:2781] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:52:43.597954 22165 raft_consensus.cc:482] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:52:43.597996 22165 raft_consensus.cc:3037] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:43.599044 22165 raft_consensus.cc:504] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:52:43.599500 22165 leader_election.cc:288] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [CANDIDATE]: Term 1 election: Requested vote from peers b396f34c2db746d4b50c39834982262d (127.6.31.129:40935), 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909)
I20260927 10:52:43.599768 21826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66a2085e8d24137af52f9ac724f74a0" candidate_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b396f34c2db746d4b50c39834982262d"
I20260927 10:52:43.599915 21826 raft_consensus.cc:3037] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:43.600226 22094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66a2085e8d24137af52f9ac724f74a0" candidate_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d60676740dc4804b2882521469db5c5"
I20260927 10:52:43.600365 22094 raft_consensus.cc:3037] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:52:43.601274 21826 raft_consensus.cc:2454] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edcf0fb871243b5a5fc3c0816c3040d in term 1.
I20260927 10:52:43.601651 21896 leader_election.cc:302] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edcf0fb871243b5a5fc3c0816c3040d, b396f34c2db746d4b50c39834982262d; no voters:
I20260927 10:52:43.601704 22094 raft_consensus.cc:2454] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edcf0fb871243b5a5fc3c0816c3040d in term 1.
I20260927 10:52:43.601842 22165 raft_consensus.cc:2781] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:52:43.602113 22165 raft_consensus.cc:686] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 LEADER]: Becoming Leader. State: Replica: 2edcf0fb871243b5a5fc3c0816c3040d, State: Running, Role: LEADER
W20260927 10:52:43.602366 21873 tablet.cc:2369] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:52:43.602532 22165 consensus_queue.cc:237] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:52:43.605808 21681 catalog_manager.cc:5495] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130). New cstate: current_term: 1 leader_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } health_report { overall_health: UNKNOWN } } }
I20260927 10:52:43.656044 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.652529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.6.31.130:33687)
0927 10:52:43.652653 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:52:43.652656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:52:43.652669 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:52:43.652805 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:52:43.652807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:52:43.652862 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:52:43.652982 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:52:43.653179 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.653667 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.654477 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:52:43.654675 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:52:43.654752 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:52:43.655163 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:52:43.655583 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:52:43.655683 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:52:43.655836 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 10:52:43.655887 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:52:43.662088 22094 raft_consensus.cc:1261] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [term 1 FOLLOWER]: Refusing update from remote peer 2edcf0fb871243b5a5fc3c0816c3040d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:43.662088 21826 raft_consensus.cc:1261] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Refusing update from remote peer 2edcf0fb871243b5a5fc3c0816c3040d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:52:43.662832 22167 consensus_queue.cc:1035] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:43.663012 22165 consensus_queue.cc:1035] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:52:43.670094 22170 mvcc.cc:205] Tried to move back new op lower bound from 7333914065552547840 to 7333914065318682624. Current Snapshot: MvccSnapshot[applied={T|T < 7333914065552547840}]
I20260927 10:52:43.678638 21680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:52:43.678823 21680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:52:43.681196 21680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Sending DeleteTablet for 3 replicas of tablet c66a2085e8d24137af52f9ac724f74a0
I20260927 10:52:43.681970 21940 tablet_service.cc:1514] Processing DeleteTablet for tablet c66a2085e8d24137af52f9ac724f74a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:37416
I20260927 10:52:43.682052 22074 tablet_service.cc:1514] Processing DeleteTablet for tablet c66a2085e8d24137af52f9ac724f74a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:44964
I20260927 10:52:43.682060 21806 tablet_service.cc:1514] Processing DeleteTablet for tablet c66a2085e8d24137af52f9ac724f74a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:47586
I20260927 10:52:43.682564 22182 tablet_replica.cc:330] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: stopping tablet replica
I20260927 10:52:43.682780 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 21743
I20260927 10:52:43.682906 22182 raft_consensus.cc:2229] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:52:43.687302 22180 tablet_replica.cc:330] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: stopping tablet replica
I20260927 10:52:43.687335 22181 tablet_replica.cc:330] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: stopping tablet replica
I20260927 10:52:43.687640 22181 raft_consensus.cc:2229] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:52:43.687700 22180 raft_consensus.cc:2229] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:52:43.688019 22181 raft_consensus.cc:2258] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:52:43.688238 22180 raft_consensus.cc:2258] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 10:52:43.688802 21668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:40935 (error 108)
I20260927 10:52:43.689654 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 21877
I20260927 10:52:43.689678 22180 ts_tablet_manager.cc:1905] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:52:43.689682 22181 ts_tablet_manager.cc:1905] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:52:43.692157 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.691785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:40935 (local address 127.0.0.1:47596)
0927 10:52:43.691946 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:43.692067 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 10:52:43.692350 21668 catalog_manager.cc:4584] TS b396f34c2db746d4b50c39834982262d (127.6.31.129:40935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c66a2085e8d24137af52f9ac724f74a0: Network error: Client connection negotiation failed: client connection to 127.6.31.129:40935: connect: Connection refused (error 111)
I20260927 10:52:43.692804 22181 ts_tablet_manager.cc:1918] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:52:43.692884 22181 log.cc:1196] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/c66a2085e8d24137af52f9ac724f74a0
I20260927 10:52:43.693277 22181 ts_tablet_manager.cc:1939] T c66a2085e8d24137af52f9ac724f74a0 P 9d60676740dc4804b2882521469db5c5: Deleting consensus metadata
I20260927 10:52:43.694440 21666 catalog_manager.cc:4841] TS 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909): tablet c66a2085e8d24137af52f9ac724f74a0 (table test-table [id=7dd200016c27408f96461f82609ab73c]) successfully deleted
I20260927 10:52:43.696381 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 22011
I20260927 10:52:43.696718 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:52:43.696375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:33687 (local address 127.0.0.1:37434)
0927 10:52:43.696549 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 10:52:43.696610 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.130:33687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 10:52:43.701860 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 21649
I20260927 10:53:03.708570 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35191
--webserver_interface=127.6.31.190
--webserver_port=43041
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:03.820039 22184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:03.820334 22184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:03.820377 22184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:03.824491 22184 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:53:03.824551 22184 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:53:03.824569 22184 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:53:03.824584 22184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:03.824601 22184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:03.824659 22184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:03.824692 22184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:03.824713 22184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:03.830122 22184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=43041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:03.831303 22184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:03.832448 22184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:03.846529 22193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:03.846510 22190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:03.846603 22184 server_base.cc:1034] running on GCE node
W20260927 10:53:03.846534 22191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:03.847497 22184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:03.848731 22184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:03.849929 22184 hybrid_clock.cc:648] HybridClock initialized: now 1790506383849916 us; error 29 us; skew 500 ppm
Sep 27 10:53:03 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:03.853087 22184 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:53:03.854125 22184 webserver.cc:460] Webserver started at http://127.6.31.190:43041/ using document root <none> and password file <none>
I20260927 10:53:03.854739 22184 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:03.854858 22184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:03.859141 22184 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:53:03.861438 22200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:03.862825 22184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:03.863068 22184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "72c8ad9989db4b69b587ed6fdca6794d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "bb71cfb47a8220966a8f9805b5317b57"
server_key_iv: "57c96abbe08a365e336a523509b7a604"
server_key_version: "encryptionkey@0"
I20260927 10:53:03.863792 22184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:03.877662 22184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:03.881630 22184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:03.881896 22184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:03.889937 22184 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:35191
I20260927 10:53:03.890215 22252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:35191 every 8 connection(s)
I20260927 10:53:03.891281 22184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:03.895361 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 22184
I20260927 10:53:03.895835 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:40935
--local_ip_for_outbound_sockets=127.6.31.129
--tserver_master_addrs=127.6.31.190:35191
--webserver_port=35433
--webserver_interface=127.6.31.129
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:53:03.895890 22253 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:53:03.898204 22253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Bootstrap starting.
I20260927 10:53:03.915637 22253 log.cc:824] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:03.924435 22253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:53:03.924878 22253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Bootstrap complete.
I20260927 10:53:03.928992 22253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:53:03.929381 22253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c8ad9989db4b69b587ed6fdca6794d, State: Initialized, Role: FOLLOWER
I20260927 10:53:03.929854 22253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:53:03.930007 22253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:03.930122 22253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:03.930226 22253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:53:03.932380 22253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:53:03.932721 22253 leader_election.cc:302] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72c8ad9989db4b69b587ed6fdca6794d; no voters:
I20260927 10:53:03.933100 22253 leader_election.cc:288] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:53:03.933269 22257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:53:03.933598 22257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 2 LEADER]: Becoming Leader. State: Replica: 72c8ad9989db4b69b587ed6fdca6794d, State: Running, Role: LEADER
I20260927 10:53:03.933961 22257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:53:03.934410 22253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:03.935498 22259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72c8ad9989db4b69b587ed6fdca6794d. Latest consensus state: current_term: 2 leader_uuid: "72c8ad9989db4b69b587ed6fdca6794d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } } }
I20260927 10:53:03.935555 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "72c8ad9989db4b69b587ed6fdca6794d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } } }
I20260927 10:53:03.935634 22259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:03.935653 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:03.935952 22266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:03.941280 22266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:03.942046 22266 catalog_manager.cc:1259] Loaded cluster ID: d9b8e1780aef4094a9e983d69e974905
I20260927 10:53:03.942111 22266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:03.944520 22266 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:03.945196 22266 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Loaded TSK: 0
I20260927 10:53:03.957648 22266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Generated new TSK 1
I20260927 10:53:03.957948 22266 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:53:04.017014 22255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:04.017283 22255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:04.017326 22255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:04.023161 22255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:04.023267 22255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:04.023373 22255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:53:04.030565 22255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:40935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=35433
--tserver_master_addrs=127.6.31.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:04.031710 22255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:04.032838 22255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:04.042872 22283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:04.042892 22281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:04.043174 22255 server_base.cc:1034] running on GCE node
W20260927 10:53:04.043315 22280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:04.043787 22255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:04.044559 22255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:04.045763 22255 hybrid_clock.cc:648] HybridClock initialized: now 1790506384045739 us; error 32 us; skew 500 ppm
Sep 27 10:53:04 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:04.049077 22255 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:53:04.049996 22255 webserver.cc:460] Webserver started at http://127.6.31.129:35433/ using document root <none> and password file <none>
I20260927 10:53:04.050585 22255 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:04.050699 22255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:04.054768 22255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:04.057106 22290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:04.058457 22255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 10:53:04.058627 22255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "b396f34c2db746d4b50c39834982262d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "c3e2279462b5f5e75b93e6ac9c9ea936"
server_key_iv: "21b9a747d43dbb386cb08029eb356138"
server_key_version: "encryptionkey@0"
I20260927 10:53:04.059266 22255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:04.076105 22255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:04.079696 22255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:04.079909 22255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:04.080533 22255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:04.081717 22297 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:53:04.085795 22255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:53:04.085860 22255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 10:53:04.085907 22255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:53:04.087674 22255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:53:04.087731 22255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:04.087985 22297 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Bootstrap starting.
I20260927 10:53:04.099355 22255 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:40935
I20260927 10:53:04.099371 22404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:40935 every 8 connection(s)
I20260927 10:53:04.100595 22255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:04.103315 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 22255
I20260927 10:53:04.103794 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:33687
--local_ip_for_outbound_sockets=127.6.31.130
--tserver_master_addrs=127.6.31.190:35191
--webserver_port=32925
--webserver_interface=127.6.31.130
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:53:04.109972 22297 log.cc:824] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:53:04 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:04.117921 22297 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:53:04.118418 22297 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Bootstrap complete.
I20260927 10:53:04.119755 22297 ts_tablet_manager.cc:1397] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20260927 10:53:04.121294 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.102708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:48875 (local address 127.6.31.190:35191)
0927 10:53:04.105299 (+ 2591us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.105325 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.105376 (+ 51us) server_negotiation.cc:407] Connection header received
0927 10:53:04.105593 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.105602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.105880 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.106194 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:04.107625 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.109163 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.111145 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.111852 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.112041 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.115246 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:04.115256 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:04.115274 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:04.115325 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:04.117488 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.118159 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.118166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.118171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.118353 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.118727 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.118730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.118732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.119213 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:04.119340 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.120053 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.120372 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2407,"thread_start_us":129,"threads_started":1}
I20260927 10:53:04.122722 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.102753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:48875)
0927 10:53:04.103359 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:04.103514 (+ 155us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:04.104803 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:04.106240 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:04.106247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:04.106732 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:04.107398 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.107421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.109346 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.109352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.110964 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.110977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.112030 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.112037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.112286 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.113051 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:04.113078 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:04.114995 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:04.117631 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.117637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.117648 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.117999 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.118455 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.118459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.118460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.118619 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.119332 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:04.119341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:04.119663 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 10:53:04.121568 (+ 1905us) client_negotiation.cc:240] Negotiation successful
0927 10:53:04.121896 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":173,"threads_started":1}
I20260927 10:53:04.122716 22297 raft_consensus.cc:348] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:53:04.123284 22297 raft_consensus.cc:729] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b396f34c2db746d4b50c39834982262d, State: Initialized, Role: FOLLOWER
I20260927 10:53:04.123754 22297 consensus_queue.cc:260] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:53:04.124761 22297 ts_tablet_manager.cc:1428] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:53:04.125463 22405 heartbeater.cc:347] Connected to a master server at 127.6.31.190:35191
I20260927 10:53:04.125698 22405 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:04.126281 22405 heartbeater.cc:511] Master 127.6.31.190:35191 requested a full tablet report, sending...
I20260927 10:53:04.128711 22217 ts_manager.cc:194] Registered new tserver with Master: b396f34c2db746d4b50c39834982262d (127.6.31.129:40935)
I20260927 10:53:04.132077 22217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:48875
I20260927 10:53:04.134907 22405 heartbeater.cc:503] Master 127.6.31.190:35191 was elected leader, sending a full tablet report...
Sep 27 10:53:04 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.129@KRBTEST.COM
I20260927 10:53:04.142905 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.131536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.6.31.129:40935)
0927 10:53:04.131947 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.131960 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.132430 (+ 470us) server_negotiation.cc:407] Connection header received
0927 10:53:04.132711 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.132717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.132966 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.133111 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:04.133884 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.134905 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.136275 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.136767 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.136866 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.139374 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:04.139384 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:04.139398 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:04.139426 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:04.140993 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.141493 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.141500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.141503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.141584 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.141859 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.141864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.141866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.142101 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:04.142239 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.142570 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.142732 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":115,"threads_started":1}
I20260927 10:53:04.142905 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.131436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:40935 (local address 127.0.0.1:44080)
0927 10:53:04.132274 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:04.132344 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:04.132539 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:04.133129 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:04.133134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:04.133605 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:04.133738 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.133747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.135039 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.135043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.136155 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.136184 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.136889 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.136892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.137004 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.137571 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:04.137586 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:04.139229 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:04.141126 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.141134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.141140 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.141377 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.141681 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.141684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.141686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.141771 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.142224 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:04.142230 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:04.142353 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 10:53:04.142491 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 10:53:04.142696 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":114,"threads_started":1}
I20260927 10:53:04.144186 22339 tablet_service.cc:1514] Processing DeleteTablet for tablet c66a2085e8d24137af52f9ac724f74a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:44080
I20260927 10:53:04.145159 22414 tablet_replica.cc:330] stopping tablet replica
I20260927 10:53:04.145517 22414 raft_consensus.cc:2229] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:04.145787 22414 raft_consensus.cc:2258] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:04.146884 22414 ts_tablet_manager.cc:1905] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:04.151572 22414 ts_tablet_manager.cc:1918] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:04.151679 22414 log.cc:1196] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal/wals/c66a2085e8d24137af52f9ac724f74a0
I20260927 10:53:04.151944 22414 ts_tablet_manager.cc:1939] T c66a2085e8d24137af52f9ac724f74a0 P b396f34c2db746d4b50c39834982262d: Deleting consensus metadata
I20260927 10:53:04.153910 22204 catalog_manager.cc:4841] TS b396f34c2db746d4b50c39834982262d (127.6.31.129:40935): tablet c66a2085e8d24137af52f9ac724f74a0 (table test-table [id=7dd200016c27408f96461f82609ab73c]) successfully deleted
W20260927 10:53:04.227828 22409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:04.228093 22409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:04.228138 22409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:04.232386 22409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:04.232468 22409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:04.232587 22409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:53:04.238230 22409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:33687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=32925
--tserver_master_addrs=127.6.31.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:04.239526 22409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:04.240492 22409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:04.250285 22419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:04.250321 22420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:04.250300 22422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:04.250514 22409 server_base.cc:1034] running on GCE node
I20260927 10:53:04.250873 22409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:04.251529 22409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:04.252723 22409 hybrid_clock.cc:648] HybridClock initialized: now 1790506384252712 us; error 31 us; skew 500 ppm
Sep 27 10:53:04 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:04.255474 22409 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:53:04.256275 22409 webserver.cc:460] Webserver started at http://127.6.31.130:32925/ using document root <none> and password file <none>
I20260927 10:53:04.256839 22409 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:04.256910 22409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:04.260877 22409 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:04.263337 22429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:04.264711 22409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:04.264837 22409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "2edcf0fb871243b5a5fc3c0816c3040d"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "1f0c1757baa66a47c6399d21d3be8a8b"
server_key_iv: "fad6dc80056b887d71346d4561b34250"
server_key_version: "encryptionkey@0"
I20260927 10:53:04.265379 22409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:04.291356 22409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:04.294691 22409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:04.294909 22409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:04.295588 22409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:04.296663 22436 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:53:04.300773 22409 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:53:04.300835 22409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 10:53:04.300915 22409 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:53:04.302594 22409 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:53:04.302637 22409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:53:04.302907 22436 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Bootstrap starting.
I20260927 10:53:04.308784 22436 log.cc:824] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:04.312704 22409 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:33687
I20260927 10:53:04.312765 22544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:33687 every 8 connection(s)
I20260927 10:53:04.313545 22409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:04.316681 22436 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:53:04.317075 22436 tablet_bootstrap.cc:492] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Bootstrap complete.
I20260927 10:53:04.317427 22436 ts_tablet_manager.cc:1397] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260927 10:53:04.320065 22436 raft_consensus.cc:348] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:53:04.320477 22436 raft_consensus.cc:729] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edcf0fb871243b5a5fc3c0816c3040d, State: Initialized, Role: FOLLOWER
I20260927 10:53:04.320947 22436 consensus_queue.cc:260] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } }
I20260927 10:53:04.321635 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 22409
I20260927 10:53:04.322018 22436 ts_tablet_manager.cc:1428] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:53:04.322132 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:38909
--local_ip_for_outbound_sockets=127.6.31.131
--tserver_master_addrs=127.6.31.190:35191
--webserver_port=38443
--webserver_interface=127.6.31.131
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:53:04 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:04.329252 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.315408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:57985 (local address 127.6.31.190:35191)
0927 10:53:04.315605 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.315610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.316201 (+ 591us) server_negotiation.cc:407] Connection header received
0927 10:53:04.317217 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.317221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.317287 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.317395 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:04.318555 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.319380 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.320465 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.320811 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.320909 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.324418 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:04.324422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:04.324424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:04.324452 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:04.326267 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.327448 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.327454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.327458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.327571 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.327926 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.327932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.327936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.328215 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:04.328335 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.328844 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.329009 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:04.330086 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.315537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:57985)
0927 10:53:04.316036 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:04.316087 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:04.316989 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:04.317561 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:04.317567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:04.317891 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:04.318393 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.318406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.319519 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.319523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.320344 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.320353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.320934 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.320937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.321136 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.322218 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:04.322245 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:04.324208 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:04.326489 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.326495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.326507 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.327013 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.327677 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.327679 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.327681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.327825 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.328329 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:04.328334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:04.328589 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 10:53:04.328765 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 10:53:04.329134 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260927 10:53:04.331447 22545 heartbeater.cc:347] Connected to a master server at 127.6.31.190:35191
I20260927 10:53:04.331736 22545 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:04.332360 22545 heartbeater.cc:511] Master 127.6.31.190:35191 requested a full tablet report, sending...
I20260927 10:53:04.334532 22217 ts_manager.cc:194] Registered new tserver with Master: 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687)
I20260927 10:53:04.335966 22217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:57985
I20260927 10:53:04.338272 22545 heartbeater.cc:503] Master 127.6.31.190:35191 was elected leader, sending a full tablet report...
Sep 27 10:53:04 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.130@KRBTEST.COM
I20260927 10:53:04.344696 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.335648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:33687 (local address 127.0.0.1:46602)
0927 10:53:04.335769 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:04.335784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:04.335869 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:04.336701 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:04.336704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:04.336926 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:04.337006 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.337012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.337956 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.337962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.338544 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.338552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.339258 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.339261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.339353 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.339853 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:04.339866 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:04.341150 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:04.342969 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.342974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.342976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.343288 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.343660 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.343663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.343665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.343752 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.344177 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:04.344180 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:04.344272 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 10:53:04.344390 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 10:53:04.344530 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 10:53:04.344812 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.335793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.6.31.130:33687)
0927 10:53:04.336191 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.336203 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.336227 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:53:04.336325 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.336331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.336564 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.336724 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:04.337118 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.337842 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.338656 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.339147 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.339260 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.341280 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:04.341295 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:04.341304 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:04.341330 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:04.342844 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.343401 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.343409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.343411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.343548 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.343841 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.343846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.343848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.344072 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:04.344210 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.344480 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.344645 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":136,"threads_started":1}
I20260927 10:53:04.345706 22478 tablet_service.cc:1514] Processing DeleteTablet for tablet c66a2085e8d24137af52f9ac724f74a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:52:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260927 10:53:04.346765 22551 tablet_replica.cc:330] stopping tablet replica
I20260927 10:53:04.347136 22551 raft_consensus.cc:2229] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:04.347396 22551 raft_consensus.cc:2258] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:04.348325 22551 ts_tablet_manager.cc:1905] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:04.351992 22551 ts_tablet_manager.cc:1918] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:04.352104 22551 log.cc:1196] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/c66a2085e8d24137af52f9ac724f74a0
I20260927 10:53:04.352372 22551 ts_tablet_manager.cc:1939] T c66a2085e8d24137af52f9ac724f74a0 P 2edcf0fb871243b5a5fc3c0816c3040d: Deleting consensus metadata
I20260927 10:53:04.353899 22204 catalog_manager.cc:4841] TS 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687): tablet c66a2085e8d24137af52f9ac724f74a0 (table test-table [id=7dd200016c27408f96461f82609ab73c]) successfully deleted
W20260927 10:53:04.439929 22549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:04.440182 22549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:04.440261 22549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:04.444410 22549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:04.444468 22549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:04.444554 22549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:53:04.450133 22549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:38909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=38443
--tserver_master_addrs=127.6.31.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:04.451253 22549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:04.452263 22549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:04.461912 22556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:04.461912 22559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:04.461964 22549 server_base.cc:1034] running on GCE node
W20260927 10:53:04.461894 22557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:04.462453 22549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:04.463135 22549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:04.464323 22549 hybrid_clock.cc:648] HybridClock initialized: now 1790506384464306 us; error 28 us; skew 500 ppm
Sep 27 10:53:04 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:04.467154 22549 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:53:04.468065 22549 webserver.cc:460] Webserver started at http://127.6.31.131:38443/ using document root <none> and password file <none>
I20260927 10:53:04.468626 22549 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:04.468698 22549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:04.472577 22549 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:53:04.474594 22566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:04.475804 22549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:04.475924 22549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "9d60676740dc4804b2882521469db5c5"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "ccbbac155d94e55cf458ef1786ad7677"
server_key_iv: "74838119368e9cd3a40d8a5c44a03348"
server_key_version: "encryptionkey@0"
I20260927 10:53:04.476468 22549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:04.497119 22549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:04.500494 22549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:04.500705 22549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:04.501296 22549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:04.502104 22549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:04.502151 22549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:04.502218 22549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:04.502259 22549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:04.511317 22549 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:38909
I20260927 10:53:04.511409 22679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:38909 every 8 connection(s)
I20260927 10:53:04.512410 22549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:53:04 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:04.521932 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 22549
I20260927 10:53:04.525760 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.514221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:39285 (local address 127.6.31.190:35191)
0927 10:53:04.514362 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.514366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.515670 (+ 1304us) server_negotiation.cc:407] Connection header received
0927 10:53:04.516488 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.516491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.516539 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.516612 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:04.517698 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.518210 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.519268 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.519477 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.519561 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.521763 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:04.521768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:04.521771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:04.521803 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:04.523781 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.524341 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.524344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.524346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.524431 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.524730 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.524733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.524735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.524927 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:04.525018 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.525558 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.525645 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:04.526744 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.514599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:39285)
0927 10:53:04.515522 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:04.515562 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:04.516312 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:04.516768 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:04.516774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:04.517133 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:04.517573 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.517585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.518310 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.518312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.519167 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.519175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.519598 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.519601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.519727 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.520236 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:04.520251 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:04.521603 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:04.523886 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.523894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.523908 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.524260 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.524518 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:04.524521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:04.524522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:04.524642 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:04.525035 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:04.525039 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:04.525316 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 10:53:04.525512 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 10:53:04.525811 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":610,"threads_started":1}
I20260927 10:53:04.528043 22680 heartbeater.cc:347] Connected to a master server at 127.6.31.190:35191
I20260927 10:53:04.528263 22680 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:04.528724 22680 heartbeater.cc:511] Master 127.6.31.190:35191 requested a full tablet report, sending...
I20260927 10:53:04.529716 22217 ts_manager.cc:194] Registered new tserver with Master: 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909)
I20260927 10:53:04.530256 22217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:39285
I20260927 10:53:04.531590 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.522945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.6.31.190:35191)
0927 10:53:04.523230 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.523235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.523285 (+ 50us) server_negotiation.cc:407] Connection header received
0927 10:53:04.523389 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.523392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.523435 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.523531 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:04.524338 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.525074 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.526178 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.526387 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.526460 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.528148 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:04.528153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:04.528156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:04.528181 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:04.529935 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.530360 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.530364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.530368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.530454 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.530769 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.530771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.530772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.530927 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:04.531168 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.531317 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.531428 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":103,"threads_started":1}
I20260927 10:53:04.533252 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:04.537562 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.533840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.6.31.190:35191)
0927 10:53:04.534062 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.534067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.534208 (+ 141us) server_negotiation.cc:407] Connection header received
0927 10:53:04.534384 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.534387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.534535 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.534650 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:04.534875 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.535440 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.536373 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.536611 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.536721 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.537031 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:04.537150 (+ 119us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 10:53:04.537398 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53848: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 10:53:04.537631 22684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53848: token expired
W20260927 10:53:04.538740 22145 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.31.190:35191)
I20260927 10:53:04.547086 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.539145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.6.31.190:35191)
0927 10:53:04.539249 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.539252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.539266 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:53:04.539407 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.539409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.539441 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.539519 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:04.540379 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.540939 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.542267 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.542461 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.542544 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.544349 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:04.544352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:04.544354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:04.544395 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:04.545696 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.546059 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.546061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.546063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.546128 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:04.546369 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:04.546371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:04.546372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:04.546536 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:04.546630 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.546816 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.546898 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:53:04.549402 22217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:04.552767 22217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:04.560554 22339 tablet_service.cc:1467] Processing CreateTablet for tablet f51cd3e8e2cc4bc0956b1b408723621e (DEFAULT_TABLE table=test-table [id=18c334090f2e4d93a7c26cb4f6185c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:04.560554 22478 tablet_service.cc:1467] Processing CreateTablet for tablet f51cd3e8e2cc4bc0956b1b408723621e (DEFAULT_TABLE table=test-table [id=18c334090f2e4d93a7c26cb4f6185c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:04.561182 22478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51cd3e8e2cc4bc0956b1b408723621e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:04.561182 22339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51cd3e8e2cc4bc0956b1b408723621e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:04.564102 22297 tablet_bootstrap.cc:492] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Bootstrap starting.
I20260927 10:53:04.564409 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.559637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38909 (local address 127.0.0.1:45066)
0927 10:53:04.559775 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:04.559792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:04.559883 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:04.560744 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:04.560747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:04.560763 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:04.560864 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.560874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.562043 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.562046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.562909 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.562919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.563948 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.563952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.564041 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.564083 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 10:53:04.564169 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 10:53:04.564250 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 10:53:04.564422 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.559832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.6.31.131:38909)
0927 10:53:04.560302 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.560311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.560329 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:53:04.560397 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.560401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.560571 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.560743 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:04.560962 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.561919 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.563037 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.563819 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.563939 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.564102 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.564199 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.564288 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":165,"threads_started":1}
I20260927 10:53:04.564656 22436 tablet_bootstrap.cc:492] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: Bootstrap starting.
I20260927 10:53:04.565016 22297 tablet_bootstrap.cc:654] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:04.565690 22436 tablet_bootstrap.cc:654] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:04.566859 22297 tablet_bootstrap.cc:492] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: No bootstrap required, opened a new log
I20260927 10:53:04.566949 22297 ts_tablet_manager.cc:1397] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260927 10:53:04.567288 22436 tablet_bootstrap.cc:492] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: No bootstrap required, opened a new log
I20260927 10:53:04.567373 22436 ts_tablet_manager.cc:1397] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:04.567462 22297 raft_consensus.cc:348] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:04.567624 22297 raft_consensus.cc:374] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:04.567674 22297 raft_consensus.cc:729] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b396f34c2db746d4b50c39834982262d, State: Initialized, Role: FOLLOWER
I20260927 10:53:04.567803 22297 consensus_queue.cc:260] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:04.567834 22436 raft_consensus.cc:348] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:04.567930 22436 raft_consensus.cc:374] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:04.567978 22436 raft_consensus.cc:729] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edcf0fb871243b5a5fc3c0816c3040d, State: Initialized, Role: FOLLOWER
I20260927 10:53:04.568099 22436 consensus_queue.cc:260] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:04.567807 22614 tablet_service.cc:1467] Processing CreateTablet for tablet f51cd3e8e2cc4bc0956b1b408723621e (DEFAULT_TABLE table=test-table [id=18c334090f2e4d93a7c26cb4f6185c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:04.568373 22436 ts_tablet_manager.cc:1428] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:53:04.568534 22297 ts_tablet_manager.cc:1428] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 10:53:04.568962 22614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51cd3e8e2cc4bc0956b1b408723621e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:04.576009 22692 tablet_bootstrap.cc:492] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: Bootstrap starting.
I20260927 10:53:04.578540 22692 tablet_bootstrap.cc:654] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:04.579347 22692 log.cc:824] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:04.581400 22692 tablet_bootstrap.cc:492] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: No bootstrap required, opened a new log
I20260927 10:53:04.581634 22692 ts_tablet_manager.cc:1397] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:53:04.584595 22692 raft_consensus.cc:348] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:04.584831 22692 raft_consensus.cc:374] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:04.584865 22692 raft_consensus.cc:729] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d60676740dc4804b2882521469db5c5, State: Initialized, Role: FOLLOWER
I20260927 10:53:04.585261 22692 consensus_queue.cc:260] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:04.585919 22680 heartbeater.cc:503] Master 127.6.31.190:35191 was elected leader, sending a full tablet report...
I20260927 10:53:04.586129 22692 ts_tablet_manager.cc:1428] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260927 10:53:04.601778 22406 tablet.cc:2369] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:04.763420 22681 tablet.cc:2369] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:04.769752 22411 raft_consensus.cc:482] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:04.770030 22411 raft_consensus.cc:504] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:04.771341 22411 leader_election.cc:288] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909), 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687)
I20260927 10:53:04.775400 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.771729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:52565 (local address 127.6.31.131:38909)
0927 10:53:04.771862 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.771866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.771978 (+ 112us) server_negotiation.cc:407] Connection header received
0927 10:53:04.772180 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.772186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.772241 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.772309 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:04.772606 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.773166 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.774375 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.774784 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.774876 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.774916 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.775181 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.775263 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:53:04.775404 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.771951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:35263 (local address 127.6.31.130:33687)
0927 10:53:04.772072 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.772076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.772095 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:53:04.772248 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.772251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.772343 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.772440 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:04.772746 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.773337 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.774421 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.774882 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.774999 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.775192 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.775229 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.775293 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:53:04.775429 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.771567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38909 (local address 127.6.31.129:52565)
0927 10:53:04.771868 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:04.771884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:04.772031 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:04.772344 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:04.772348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:04.772365 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:04.772491 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.772499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.773285 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.773289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.774252 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.774261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.774905 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.774912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.775006 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.775063 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 10:53:04.775164 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 10:53:04.775252 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":48,"thread_start_us":110,"threads_started":1}
I20260927 10:53:04.775460 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.771618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:33687 (local address 127.6.31.129:35263)
0927 10:53:04.772000 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:04.772015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:04.772129 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:04.772464 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:04.772466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:04.772481 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:04.772597 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.772603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.773451 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.773454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.774263 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:04.774271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.775012 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.775015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:04.775077 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.775152 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 10:53:04.775217 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 10:53:04.775281 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":41,"thread_start_us":80,"threads_started":1}
I20260927 10:53:04.776096 22498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51cd3e8e2cc4bc0956b1b408723621e" candidate_uuid: "b396f34c2db746d4b50c39834982262d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" is_pre_election: true
I20260927 10:53:04.776132 22634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51cd3e8e2cc4bc0956b1b408723621e" candidate_uuid: "b396f34c2db746d4b50c39834982262d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d60676740dc4804b2882521469db5c5" is_pre_election: true
I20260927 10:53:04.776450 22498 raft_consensus.cc:2454] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b396f34c2db746d4b50c39834982262d in term 0.
I20260927 10:53:04.776487 22634 raft_consensus.cc:2454] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b396f34c2db746d4b50c39834982262d in term 0.
I20260927 10:53:04.776993 22294 leader_election.cc:302] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2edcf0fb871243b5a5fc3c0816c3040d, 9d60676740dc4804b2882521469db5c5, b396f34c2db746d4b50c39834982262d; no voters:
I20260927 10:53:04.777287 22411 raft_consensus.cc:2781] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:04.777388 22411 raft_consensus.cc:482] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:04.777453 22411 raft_consensus.cc:3037] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:04.778494 22411 raft_consensus.cc:504] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:04.778885 22411 leader_election.cc:288] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 1 election: Requested vote from peers 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909), 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687)
I20260927 10:53:04.779327 22498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51cd3e8e2cc4bc0956b1b408723621e" candidate_uuid: "b396f34c2db746d4b50c39834982262d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edcf0fb871243b5a5fc3c0816c3040d"
I20260927 10:53:04.779371 22634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51cd3e8e2cc4bc0956b1b408723621e" candidate_uuid: "b396f34c2db746d4b50c39834982262d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d60676740dc4804b2882521469db5c5"
I20260927 10:53:04.779459 22498 raft_consensus.cc:3037] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:04.779542 22634 raft_consensus.cc:3037] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:04.780731 22634 raft_consensus.cc:2454] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b396f34c2db746d4b50c39834982262d in term 1.
I20260927 10:53:04.780748 22498 raft_consensus.cc:2454] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b396f34c2db746d4b50c39834982262d in term 1.
I20260927 10:53:04.781090 22292 leader_election.cc:302] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2edcf0fb871243b5a5fc3c0816c3040d, 9d60676740dc4804b2882521469db5c5, b396f34c2db746d4b50c39834982262d; no voters:
I20260927 10:53:04.781283 22411 raft_consensus.cc:2781] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:04.781497 22411 raft_consensus.cc:686] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 LEADER]: Becoming Leader. State: Replica: b396f34c2db746d4b50c39834982262d, State: Running, Role: LEADER
I20260927 10:53:04.781790 22411 consensus_queue.cc:237] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:04.784819 22216 catalog_manager.cc:5495] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d reported cstate change: term changed from 0 to 1, leader changed from <none> to b396f34c2db746d4b50c39834982262d (127.6.31.129). New cstate: current_term: 1 leader_uuid: "b396f34c2db746d4b50c39834982262d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } health_report { overall_health: HEALTHY } } }
I20260927 10:53:04.805531 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.802217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.6.31.129:40935)
0927 10:53:04.802473 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:04.802480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:04.802510 (+ 30us) server_negotiation.cc:407] Connection header received
0927 10:53:04.802579 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:04.802582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:04.802635 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:04.802737 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:04.802933 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.803532 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.804284 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:04.804502 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:04.804577 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:04.804796 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:04.805202 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:04.805287 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:04.805345 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 10:53:04.805388 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":114,"threads_started":1}
I20260927 10:53:04.809437 22498 raft_consensus.cc:1261] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Refusing update from remote peer b396f34c2db746d4b50c39834982262d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:04.809440 22634 raft_consensus.cc:1261] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [term 1 FOLLOWER]: Refusing update from remote peer b396f34c2db746d4b50c39834982262d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:04.810184 22697 consensus_queue.cc:1035] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:04.810329 22411 consensus_queue.cc:1035] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:53:04.816260 22546 tablet.cc:2369] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:04.816978 22701 mvcc.cc:205] Tried to move back new op lower bound from 7333914152172843008 to 7333914152068923392. Current Snapshot: MvccSnapshot[applied={T|T < 7333914152172843008}]
I20260927 10:53:04.818228 22702 mvcc.cc:205] Tried to move back new op lower bound from 7333914152172843008 to 7333914152068923392. Current Snapshot: MvccSnapshot[applied={T|T < 7333914152172843008}]
I20260927 10:53:04.825773 22217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:04.825971 22217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:04.828393 22217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Sending DeleteTablet for 3 replicas of tablet f51cd3e8e2cc4bc0956b1b408723621e
I20260927 10:53:04.829221 22339 tablet_service.cc:1514] Processing DeleteTablet for tablet f51cd3e8e2cc4bc0956b1b408723621e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:04 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:44080
I20260927 10:53:04.829226 22614 tablet_service.cc:1514] Processing DeleteTablet for tablet f51cd3e8e2cc4bc0956b1b408723621e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:04 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:45066
I20260927 10:53:04.829430 22478 tablet_service.cc:1514] Processing DeleteTablet for tablet f51cd3e8e2cc4bc0956b1b408723621e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:04 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260927 10:53:04.829632 22551 tablet_replica.cc:330] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: stopping tablet replica
I20260927 10:53:04.829653 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 22255
I20260927 10:53:04.829706 22712 tablet_replica.cc:330] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: stopping tablet replica
I20260927 10:53:04.829761 22551 raft_consensus.cc:2229] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:04.829887 22551 raft_consensus.cc:2258] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:04.830030 22712 raft_consensus.cc:2229] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:04.830446 22712 raft_consensus.cc:2258] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:04.831204 22551 ts_tablet_manager.cc:1905] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:04.833156 22712 ts_tablet_manager.cc:1905] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:04.833696 22551 ts_tablet_manager.cc:1918] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:04.833776 22551 log.cc:1196] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/f51cd3e8e2cc4bc0956b1b408723621e
I20260927 10:53:04.834023 22551 ts_tablet_manager.cc:1939] T f51cd3e8e2cc4bc0956b1b408723621e P 2edcf0fb871243b5a5fc3c0816c3040d: Deleting consensus metadata
I20260927 10:53:04.835189 22204 catalog_manager.cc:4841] TS 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687): tablet f51cd3e8e2cc4bc0956b1b408723621e (table test-table [id=18c334090f2e4d93a7c26cb4f6185c77]) successfully deleted
I20260927 10:53:04.836059 22712 ts_tablet_manager.cc:1918] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:04.836135 22712 log.cc:1196] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/f51cd3e8e2cc4bc0956b1b408723621e
I20260927 10:53:04.836445 22712 ts_tablet_manager.cc:1939] T f51cd3e8e2cc4bc0956b1b408723621e P 9d60676740dc4804b2882521469db5c5: Deleting consensus metadata
W20260927 10:53:04.837349 22204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:40935 (error 108)
I20260927 10:53:04.837481 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 22409
I20260927 10:53:04.837777 22202 catalog_manager.cc:4841] TS 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909): tablet f51cd3e8e2cc4bc0956b1b408723621e (table test-table [id=18c334090f2e4d93a7c26cb4f6185c77]) successfully deleted
I20260927 10:53:04.839068 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:04.838771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:40935 (local address 127.0.0.1:44108)
0927 10:53:04.838901 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:04.838969 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:53:04.839277 22204 catalog_manager.cc:4584] TS b396f34c2db746d4b50c39834982262d (127.6.31.129:40935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f51cd3e8e2cc4bc0956b1b408723621e: Network error: Client connection negotiation failed: client connection to 127.6.31.129:40935: connect: Connection refused (error 111)
I20260927 10:53:04.843724 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 22549
I20260927 10:53:04.849936 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 22184
I20260927 10:53:24.857088 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35191
--webserver_interface=127.6.31.190
--webserver_port=43041
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:24.970222 22714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:24.970484 22714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:24.970562 22714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:24.974745 22714 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:53:24.974802 22714 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:53:24.974818 22714 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:53:24.974833 22714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:24.974874 22714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:24.974920 22714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:24.974952 22714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:24.974987 22714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:24.980293 22714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=43041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:24.981341 22714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:24.982347 22714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:24.991707 22714 server_base.cc:1034] running on GCE node
W20260927 10:53:24.991694 22720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:24.991715 22719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:24.991789 22722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:24.992586 22714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:24.993738 22714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:24.994937 22714 hybrid_clock.cc:648] HybridClock initialized: now 1790506404994908 us; error 44 us; skew 500 ppm
Sep 27 10:53:25 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:25.005371 22714 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:53:25.006453 22714 webserver.cc:460] Webserver started at http://127.6.31.190:43041/ using document root <none> and password file <none>
I20260927 10:53:25.007068 22714 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:25.007264 22714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:25.011371 22714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 10:53:25.013801 22729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.015218 22714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:25.015372 22714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "72c8ad9989db4b69b587ed6fdca6794d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "bb71cfb47a8220966a8f9805b5317b57"
server_key_iv: "57c96abbe08a365e336a523509b7a604"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.015944 22714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:25.043985 22714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:25.047389 22714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:25.047603 22714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:25.054680 22714 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:35191
I20260927 10:53:25.054996 22781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:35191 every 8 connection(s)
I20260927 10:53:25.055852 22714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:25.060190 22782 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:53:25.061883 22782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Bootstrap starting.
I20260927 10:53:25.064390 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 22714
I20260927 10:53:25.064872 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:40935
--local_ip_for_outbound_sockets=127.6.31.129
--tserver_master_addrs=127.6.31.190:35191
--webserver_port=35433
--webserver_interface=127.6.31.129
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:53:25.080112 22782 log.cc:824] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:25.091773 22782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:53:25.092254 22782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: Bootstrap complete.
I20260927 10:53:25.096480 22782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:53:25.096896 22782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c8ad9989db4b69b587ed6fdca6794d, State: Initialized, Role: FOLLOWER
I20260927 10:53:25.097385 22782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:53:25.097545 22782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:25.097630 22782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:25.097777 22782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:53:25.099735 22782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:53:25.100077 22782 leader_election.cc:302] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72c8ad9989db4b69b587ed6fdca6794d; no voters:
I20260927 10:53:25.100401 22782 leader_election.cc:288] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:53:25.100536 22786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:53:25.100812 22786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [term 3 LEADER]: Becoming Leader. State: Replica: 72c8ad9989db4b69b587ed6fdca6794d, State: Running, Role: LEADER
I20260927 10:53:25.101186 22786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } }
I20260927 10:53:25.101688 22782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:25.102655 22787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "72c8ad9989db4b69b587ed6fdca6794d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } } }
I20260927 10:53:25.102774 22787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:25.103607 22788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72c8ad9989db4b69b587ed6fdca6794d. Latest consensus state: current_term: 3 leader_uuid: "72c8ad9989db4b69b587ed6fdca6794d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c8ad9989db4b69b587ed6fdca6794d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35191 } } }
I20260927 10:53:25.103724 22788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:25.106734 22802 catalog_manager.cc:1259] Loaded cluster ID: d9b8e1780aef4094a9e983d69e974905
I20260927 10:53:25.106791 22802 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: loading cluster ID for follower catalog manager: success
I20260927 10:53:25.109325 22802 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72c8ad9989db4b69b587ed6fdca6794d: acquiring CA information for follower catalog manager: success
F20260927 10:53:25.110540 22802 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0442ac69ad google::LogMessage::Fail() at ??:0
@ 0x7f0442ac988a google::LogMessage::SendToLog() at ??:0
@ 0x7f0442ac64de google::LogMessage::Flush() at ??:0
@ 0x7f0442ac7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f044578a18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f044d42b989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f044d408023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f044d40854b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f044d401652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f044d4009e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f044d447541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0444d09512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f044634d6db start_thread at ??:0
@ 0x7f04415c461f clone at ??:0
W20260927 10:53:25.195801 22784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:25.196116 22784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:25.196193 22784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:25.200492 22784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:25.200577 22784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:25.200668 22784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:53:25.206065 22784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:40935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=35433
--tserver_master_addrs=127.6.31.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:25.207224 22784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:25.208243 22784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:25.218276 22823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.218277 22822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.218302 22825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:25.218824 22784 server_base.cc:1034] running on GCE node
I20260927 10:53:25.219340 22784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:25.219988 22784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:25.221176 22784 hybrid_clock.cc:648] HybridClock initialized: now 1790506405221133 us; error 61 us; skew 500 ppm
Sep 27 10:53:25 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:25.224213 22784 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:53:25.225158 22784 webserver.cc:460] Webserver started at http://127.6.31.129:35433/ using document root <none> and password file <none>
I20260927 10:53:25.225741 22784 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:25.225833 22784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:25.229987 22784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:25.232506 22832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.234043 22784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:53:25.234174 22784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "b396f34c2db746d4b50c39834982262d"
format_stamp: "Formatted at 2026-09-27 10:52:42 on dist-test-slave-c60x"
server_key: "c3e2279462b5f5e75b93e6ac9c9ea936"
server_key_iv: "21b9a747d43dbb386cb08029eb356138"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.234714 22784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:25.257019 22784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:25.260445 22784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:25.260658 22784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:25.261346 22784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:25.262476 22839 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:53:25.268368 22784 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:53:25.268450 22784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 10:53:25.268541 22784 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:53:25.270282 22784 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:53:25.270355 22784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:53:25.270665 22839 tablet_bootstrap.cc:492] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Bootstrap starting.
I20260927 10:53:25.282673 22784 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:40935
I20260927 10:53:25.282738 22946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:40935 every 8 connection(s)
I20260927 10:53:25.284029 22784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:25.291054 22839 log.cc:824] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:25.292135 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 22784
I20260927 10:53:25.292706 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:33687
--local_ip_for_outbound_sockets=127.6.31.130
--tserver_master_addrs=127.6.31.190:35191
--webserver_port=32925
--webserver_interface=127.6.31.130
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:53:25.298848 22839 tablet_bootstrap.cc:492] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:53:25.299264 22839 tablet_bootstrap.cc:492] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Bootstrap complete.
I20260927 10:53:25.300405 22839 ts_tablet_manager.cc:1397] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.005s
I20260927 10:53:25.302755 22839 raft_consensus.cc:348] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:25.303231 22839 raft_consensus.cc:729] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b396f34c2db746d4b50c39834982262d, State: Initialized, Role: FOLLOWER
I20260927 10:53:25.303721 22839 consensus_queue.cc:260] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:25.304651 22839 ts_tablet_manager.cc:1428] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
W20260927 10:53:25.368148 22949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:25.286474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:53889)
0927 10:53:25.287713 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.287772 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:25.288778 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:25.367104 (+ 78326us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":144,"threads_started":1}
W20260927 10:53:25.368788 22947 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:35191 (0 consecutive failures): Network error: Failed to ping master at 127.6.31.190:35191: Client connection negotiation failed: client connection to 127.6.31.190:35191: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:53:25.369586 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.369298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:43433)
0927 10:53:25.369457 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.369532 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 10:53:25.370277 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.370079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:39301)
0927 10:53:25.370180 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.370233 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:53:25.370473 22947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:53:25.424955 22951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:25.425338 22951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:25.425416 22951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:25.429889 22951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:25.429955 22951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:25.430015 22951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:53:25.435750 22951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:33687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=32925
--tserver_master_addrs=127.6.31.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:25.436779 22951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:25.437816 22951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:25.447729 22958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.447738 22960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.447767 22957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:25.447958 22951 server_base.cc:1034] running on GCE node
I20260927 10:53:25.448335 22951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:25.449024 22951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:25.450237 22951 hybrid_clock.cc:648] HybridClock initialized: now 1790506405450216 us; error 30 us; skew 500 ppm
Sep 27 10:53:25 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:25.453018 22951 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:53:25.453897 22951 webserver.cc:460] Webserver started at http://127.6.31.130:32925/ using document root <none> and password file <none>
I20260927 10:53:25.454485 22951 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:25.454577 22951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:25.458559 22951 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:25.460721 22967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.461912 22951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:25.462051 22951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "2edcf0fb871243b5a5fc3c0816c3040d"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "1f0c1757baa66a47c6399d21d3be8a8b"
server_key_iv: "fad6dc80056b887d71346d4561b34250"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.462607 22951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:25.487905 22951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:25.491112 22951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:25.491345 22951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:25.491961 22951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:25.492812 22951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:25.492883 22951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.492947 22951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:25.492990 22951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.502202 22951 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:33687
I20260927 10:53:25.502228 23080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:33687 every 8 connection(s)
I20260927 10:53:25.503232 22951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:25.507201 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.505468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:38915)
0927 10:53:25.505969 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.506477 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":121,"threads_started":1}
W20260927 10:53:25.507804 23081 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:35191 (0 consecutive failures): Network error: Failed to ping master at 127.6.31.190:35191: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
I20260927 10:53:25.508519 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.508279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:38177)
0927 10:53:25.508398 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.508464 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 10:53:25.509200 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.509012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:56697)
0927 10:53:25.509111 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.509151 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:53:25.509395 23081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:53:25.513561 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 22951
I20260927 10:53:25.514003 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:38909
--local_ip_for_outbound_sockets=127.6.31.131
--tserver_master_addrs=127.6.31.190:35191
--webserver_port=38443
--webserver_interface=127.6.31.131
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:25.624835 23084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:25.625110 23084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:25.625154 23084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:25.629395 23084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:25.629467 23084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:25.629572 23084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:53:25.635449 23084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:38909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=38443
--tserver_master_addrs=127.6.31.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:25.636574 23084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:25.637627 23084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:25.647413 23090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.647375 23092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.647362 23089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:25.647809 23084 server_base.cc:1034] running on GCE node
I20260927 10:53:25.648213 23084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:25.648836 23084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:25.650017 23084 hybrid_clock.cc:648] HybridClock initialized: now 1790506405649990 us; error 38 us; skew 500 ppm
Sep 27 10:53:25 dist-test-slave-c60x krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:25.652735 23084 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:53:25.653653 23084 webserver.cc:460] Webserver started at http://127.6.31.131:38443/ using document root <none> and password file <none>
I20260927 10:53:25.654222 23084 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:25.654287 23084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:25.657989 23084 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:25.660035 23099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.661258 23084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:25.661350 23084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "9d60676740dc4804b2882521469db5c5"
format_stamp: "Formatted at 2026-09-27 10:52:43 on dist-test-slave-c60x"
server_key: "ccbbac155d94e55cf458ef1786ad7677"
server_key_iv: "74838119368e9cd3a40d8a5c44a03348"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.661870 23084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:25.676321 23084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:25.679283 23084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:25.679437 23084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:25.679958 23084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:25.680763 23084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:25.680807 23084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.680873 23084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:25.680908 23084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.689342 23084 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:38909
I20260927 10:53:25.689396 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:38909 every 8 connection(s)
I20260927 10:53:25.690346 23084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:25.694172 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.692530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:54759)
0927 10:53:25.692964 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.693448 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
W20260927 10:53:25.694631 23213 heartbeater.cc:644] Failed to heartbeat to 127.6.31.190:35191 (0 consecutive failures): Network error: Failed to ping master at 127.6.31.190:35191: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
I20260927 10:53:25.695287 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.695038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:41569)
0927 10:53:25.695192 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.695237 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 10:53:25.695910 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.695734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:39025)
0927 10:53:25.695825 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.695865 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 10:53:25.696139 23213 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:53:25.700611 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 23084
I20260927 10:53:26.371584 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:26.371161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:42845)
0927 10:53:26.371418 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:26.371473 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260927 10:53:26.510519 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:26.509993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:56401)
0927 10:53:26.510312 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:26.510398 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":109,"threads_started":1}
I20260927 10:53:26.640636 23219 raft_consensus.cc:482] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:26.640841 23219 raft_consensus.cc:504] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:26.642072 23219 leader_election.cc:288] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909), 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687)
Sep 27 10:53:26 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.130@KRBTEST.COM
Sep 27 10:53:26 dist-test-slave-c60x krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.131@KRBTEST.COM
I20260927 10:53:26.656180 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:26.642569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:35391 (local address 127.6.31.131:38909)
0927 10:53:26.642965 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:26.642979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:26.643025 (+ 46us) server_negotiation.cc:407] Connection header received
0927 10:53:26.643776 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:26.643788 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:26.644140 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:26.644435 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:26.645681 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:26.646648 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:26.647907 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:26.648352 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:26.648527 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:26.651563 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:26.651567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:26.651576 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:26.651603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:26.653413 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:26.654255 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:26.654261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:26.654266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:26.654389 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:26.654680 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:26.654684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:26.654686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:26.655152 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:26.655282 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:26.655730 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 10:53:26.655991 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":70,"threads_started":1}
I20260927 10:53:26.656172 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:26.642383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38909 (local address 127.6.31.129:35391)
0927 10:53:26.642853 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:26.642871 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:26.642983 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:26.644505 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:26.644511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:26.645022 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:26.645502 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:26.645518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:26.646798 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:26.646806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:26.647779 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:26.647788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:26.648482 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:26.648486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:26.648702 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:26.649431 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:26.649443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:26.651383 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:26.653548 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:26.653551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:26.653553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:26.653908 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:26.654485 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:26.654491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:26.654495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:26.654583 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:26.655251 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:26.655254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:26.655446 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 10:53:26.655615 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 10:53:26.655973 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":180}
I20260927 10:53:26.656302 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:26.642383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:33687 (local address 127.6.31.129:59613)
0927 10:53:26.642949 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:26.642964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:26.643106 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:26.644736 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:26.644738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:26.645014 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:26.645502 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:26.645518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:26.646802 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:26.646805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:26.647772 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:26.647781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:26.648487 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:26.648490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:26.648699 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:26.649368 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:26.649384 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:26.651016 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:26.653508 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:26.653514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:26.653528 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:26.653939 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:26.654353 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:26.654356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:26.654358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:26.654501 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:26.655206 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:26.655213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:26.655434 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 10:53:26.655615 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 10:53:26.656072 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":362,"spinlock_wait_cycles":27392,"thread_start_us":55,"threads_started":1}
I20260927 10:53:26.656810 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:26.642527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:59613 (local address 127.6.31.130:33687)
0927 10:53:26.642876 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:26.642907 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:26.643125 (+ 218us) server_negotiation.cc:407] Connection header received
0927 10:53:26.643837 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:26.643842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:26.644140 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:26.644376 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:26.645673 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:26.646639 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:26.647890 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:26.648356 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:26.648537 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:26.651284 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:26.651290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:26.651301 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:26.651330 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:26.653365 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:26.654075 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:26.654081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:26.654086 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:26.654265 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:26.654608 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:26.654611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:26.654613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:26.655062 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:26.655233 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:26.655741 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 10:53:26.656028 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":82,"threads_started":1}
W20260927 10:53:26.659036 22836 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
W20260927 10:53:26.659277 22834 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
I20260927 10:53:26.659356 22834 leader_election.cc:302] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b396f34c2db746d4b50c39834982262d; no voters: 2edcf0fb871243b5a5fc3c0816c3040d, 9d60676740dc4804b2882521469db5c5
I20260927 10:53:26.659682 23219 raft_consensus.cc:2726] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:53:26.697244 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:26.696699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:46079)
0927 10:53:26.697046 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:26.697107 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":75,"threads_started":1}
I20260927 10:53:27.372792 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:27.372347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:50349)
0927 10:53:27.372618 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:27.372677 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260927 10:53:27.511785 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:27.511367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:47997)
0927 10:53:27.511635 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:27.511691 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260927 10:53:27.698593 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:27.698060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:56293)
0927 10:53:27.698358 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:27.698421 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":114,"threads_started":1}
I20260927 10:53:28.190899 23227 raft_consensus.cc:482] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:28.191037 23227 raft_consensus.cc:504] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:28.191514 23227 leader_election.cc:288] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909), 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687)
W20260927 10:53:28.192354 22834 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
W20260927 10:53:28.192451 22836 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
I20260927 10:53:28.192564 22836 leader_election.cc:302] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b396f34c2db746d4b50c39834982262d; no voters: 2edcf0fb871243b5a5fc3c0816c3040d, 9d60676740dc4804b2882521469db5c5
I20260927 10:53:28.192731 23227 raft_consensus.cc:2726] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:53:28.374017 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:28.373567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:41405)
0927 10:53:28.373848 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:28.373908 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":78,"threads_started":1}
I20260927 10:53:28.513108 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:28.512584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:42499)
0927 10:53:28.512877 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:28.512944 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260927 10:53:28.699939 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:28.699545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:54485)
0927 10:53:28.699791 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:28.699848 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260927 10:53:29.375519 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:29.374999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:47565)
0927 10:53:29.375355 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:29.375422 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":121,"threads_started":1}
I20260927 10:53:29.514554 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:29.514085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:47245)
0927 10:53:29.514363 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:29.514432 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
I20260927 10:53:29.701392 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:29.700916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:44289)
0927 10:53:29.701226 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:29.701293 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
I20260927 10:53:30.376894 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:30.376388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:35741)
0927 10:53:30.376725 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:30.376784 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":85,"threads_started":1}
I20260927 10:53:30.515972 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:30.515515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:47553)
0927 10:53:30.515765 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:30.515824 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260927 10:53:30.702848 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:30.702421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:49345)
0927 10:53:30.702693 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:30.702755 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260927 10:53:30.903246 23237 raft_consensus.cc:482] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:30.903436 23237 raft_consensus.cc:504] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:30.903898 23237 leader_election.cc:288] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909), 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687)
W20260927 10:53:30.904770 22836 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
W20260927 10:53:30.904860 22834 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
I20260927 10:53:30.904942 22834 leader_election.cc:302] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b396f34c2db746d4b50c39834982262d; no voters: 2edcf0fb871243b5a5fc3c0816c3040d, 9d60676740dc4804b2882521469db5c5
I20260927 10:53:30.905141 23237 raft_consensus.cc:2726] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:53:31.378160 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:31.377726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:41235)
0927 10:53:31.377978 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:31.378038 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260927 10:53:31.517206 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:31.516741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:40581)
0927 10:53:31.517021 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:31.517074 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260927 10:53:31.704071 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:31.703680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:51181)
0927 10:53:31.703920 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:31.703979 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260927 10:53:32.379640 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:32.379141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:60787)
0927 10:53:32.379447 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:32.379529 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260927 10:53:32.518719 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:32.518233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:49503)
0927 10:53:32.518544 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:32.518621 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":74,"threads_started":1}
I20260927 10:53:32.705346 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:32.704871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:34237)
0927 10:53:32.705191 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:32.705251 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":73,"threads_started":1}
I20260927 10:53:33.381098 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.380555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:50901)
0927 10:53:33.380852 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.380987 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260927 10:53:33.519934 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.519514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:36393)
0927 10:53:33.519791 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.519840 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":105,"threads_started":1}
I20260927 10:53:33.706559 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.706099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:52347)
0927 10:53:33.706372 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.706430 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":93,"threads_started":1}
I20260927 10:53:34.382495 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.382046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:44831)
0927 10:53:34.382334 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.382398 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260927 10:53:34.521258 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.520779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:60683)
0927 10:53:34.521077 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.521141 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260927 10:53:34.707847 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.707434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:33935)
0927 10:53:34.707689 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.707751 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260927 10:53:35.384083 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.383587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:60925)
0927 10:53:35.383900 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:35.383970 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260927 10:53:35.522696 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.522266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:34233)
0927 10:53:35.522538 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:35.522597 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260927 10:53:35.564653 23252 raft_consensus.cc:482] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:35.564795 23252 raft_consensus.cc:504] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:35.565378 23252 leader_election.cc:288] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909), 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687)
W20260927 10:53:35.566545 22834 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
W20260927 10:53:35.566661 22836 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
I20260927 10:53:35.566731 22836 leader_election.cc:302] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b396f34c2db746d4b50c39834982262d; no voters: 2edcf0fb871243b5a5fc3c0816c3040d, 9d60676740dc4804b2882521469db5c5
I20260927 10:53:35.566939 23252 raft_consensus.cc:2726] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:53:35.709254 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.708765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:56219)
0927 10:53:35.709068 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:35.709146 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20260927 10:53:36.385465 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:36.385062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:52421)
0927 10:53:36.385318 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:36.385371 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":77,"threads_started":1}
I20260927 10:53:36.524093 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:36.523605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:52239)
0927 10:53:36.523934 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:36.523995 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260927 10:53:36.710610 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:36.710111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:46225)
0927 10:53:36.710448 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:36.710511 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
I20260927 10:53:37.386951 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:37.386427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:46659)
0927 10:53:37.386750 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:37.386837 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20260927 10:53:37.525315 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:37.524892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:56395)
0927 10:53:37.525156 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:37.525217 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260927 10:53:37.712029 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:37.711608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:55643)
0927 10:53:37.711874 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:37.711935 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260927 10:53:38.348131 23260 raft_consensus.cc:482] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:38.348300 23260 raft_consensus.cc:504] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d60676740dc4804b2882521469db5c5" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38909 } } peers { permanent_uuid: "2edcf0fb871243b5a5fc3c0816c3040d" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 33687 } } peers { permanent_uuid: "b396f34c2db746d4b50c39834982262d" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 40935 } }
I20260927 10:53:38.348739 23260 leader_election.cc:288] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909), 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687)
W20260927 10:53:38.349640 22834 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d60676740dc4804b2882521469db5c5 (127.6.31.131:38909): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
W20260927 10:53:38.349733 22836 leader_election.cc:341] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2edcf0fb871243b5a5fc3c0816c3040d (127.6.31.130:33687): Not found: Tablet not found: f51cd3e8e2cc4bc0956b1b408723621e
I20260927 10:53:38.349797 22836 leader_election.cc:302] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b396f34c2db746d4b50c39834982262d; no voters: 2edcf0fb871243b5a5fc3c0816c3040d, 9d60676740dc4804b2882521469db5c5
I20260927 10:53:38.349936 23260 raft_consensus.cc:2726] T f51cd3e8e2cc4bc0956b1b408723621e P b396f34c2db746d4b50c39834982262d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:53:38.388118 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.387705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:40519)
0927 10:53:38.387962 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:38.388018 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260927 10:53:38.526612 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.526171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:37989)
0927 10:53:38.526453 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:38.526514 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260927 10:53:38.713258 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.712829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:40515)
0927 10:53:38.713105 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:38.713164 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260927 10:53:39.389370 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:39.388951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:37185)
0927 10:53:39.389216 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:39.389276 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260927 10:53:39.527839 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:39.527426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:48263)
0927 10:53:39.527690 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:39.527745 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260927 10:53:39.714694 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:39.714267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:33651)
0927 10:53:39.714537 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:39.714597 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260927 10:53:40.390686 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.390304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.129:58181)
0927 10:53:40.390542 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.390591 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260927 10:53:40.529167 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.528737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.130:51973)
0927 10:53:40.528997 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.529066 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260927 10:53:40.706511 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 22784
I20260927 10:53:40.712908 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 22951
I20260927 10:53:40.716176 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.715747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:35191 (local address 127.6.31.131:33985)
0927 10:53:40.715996 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.716047 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260927 10:53:40.718703 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 23084
2026-09-27T10:53:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59507 ms)
[----------] 30 tests from SecurityITest (244997 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:40 dist-test-slave-c60x krb5kdc[23272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:40 dist-test-slave-c60x krb5kdc[23272](info): set up 2 sockets
Sep 27 10:53:40 dist-test-slave-c60x krb5kdc[23272](info): commencing operation
krb5kdc: starting...
W20260927 10:53:41.763518 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:53:41 dist-test-slave-c60x krb5kdc[23272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:53:41.891234 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33457
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:33457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:42.006834 23285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:42.007195 23285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:42.007244 23285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:42.011310 23285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:42.011379 23285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:42.011399 23285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:42.011415 23285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:42.011430 23285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:42.016947 23285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:33457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:33457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:42.018007 23285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:42.019006 23285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:42.030906 23290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:42.030965 23285 server_base.cc:1034] running on GCE node
W20260927 10:53:42.030887 23293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:42.030906 23291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:42.031924 23285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:42.033054 23285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:42.034229 23285 hybrid_clock.cc:648] HybridClock initialized: now 1790506422034195 us; error 49 us; skew 500 ppm
Sep 27 10:53:42 dist-test-slave-c60x krb5kdc[23272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:42.037220 23285 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:53:42.038197 23285 webserver.cc:460] Webserver started at http://127.6.31.190:46559/ using document root <none> and password file <none>
I20260927 10:53:42.038776 23285 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:42.038869 23285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:42.039156 23285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:42.041404 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "6116a552a6534fa286c6bc22898f756d"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "f5d5dda86e5b6c0fe62319c7af16f311"
server_key_iv: "e12fc791ee081f8bd499aee4f13293cc"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.042001 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "6116a552a6534fa286c6bc22898f756d"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "f5d5dda86e5b6c0fe62319c7af16f311"
server_key_iv: "e12fc791ee081f8bd499aee4f13293cc"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.045713 23285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 10:53:42.048127 23300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.049451 23285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:53:42.049629 23285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "6116a552a6534fa286c6bc22898f756d"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "f5d5dda86e5b6c0fe62319c7af16f311"
server_key_iv: "e12fc791ee081f8bd499aee4f13293cc"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.049747 23285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:42.068691 23285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:42.072105 23285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:42.072359 23285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:42.079800 23285 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:33457
I20260927 10:53:42.079820 23352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:33457 every 8 connection(s)
I20260927 10:53:42.080811 23285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:42.083729 23353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:42.088079 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 23285
I20260927 10:53:42.088227 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:42.088511 6270 external_mini_cluster.cc:1428] Setting key dffff78244714625cc0933ed853cd93b
I20260927 10:53:42.089926 23353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d: Bootstrap starting.
I20260927 10:53:42.092411 23353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:42.093258 23353 log.cc:824] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:42.095403 23353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d: No bootstrap required, opened a new log
Sep 27 10:53:42 dist-test-slave-c60x krb5kdc[23272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:42.099771 23353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6116a552a6534fa286c6bc22898f756d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33457 } }
I20260927 10:53:42.100030 23353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:42.100116 23353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6116a552a6534fa286c6bc22898f756d, State: Initialized, Role: FOLLOWER
I20260927 10:53:42.100564 23353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6116a552a6534fa286c6bc22898f756d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33457 } }
I20260927 10:53:42.100721 23353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:42.100806 23353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:42.100920 23353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:42.102093 23353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6116a552a6534fa286c6bc22898f756d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33457 } }
I20260927 10:53:42.102459 23353 leader_election.cc:302] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6116a552a6534fa286c6bc22898f756d; no voters:
I20260927 10:53:42.102829 23353 leader_election.cc:288] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:42.102967 23358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:42.103258 23358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [term 1 LEADER]: Becoming Leader. State: Replica: 6116a552a6534fa286c6bc22898f756d, State: Running, Role: LEADER
I20260927 10:53:42.103626 23358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6116a552a6534fa286c6bc22898f756d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33457 } }
I20260927 10:53:42.104183 23353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:42.104207 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.089966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.6.31.190:33457)
0927 10:53:42.090494 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.090510 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.090548 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:53:42.091295 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.091314 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.091646 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.091936 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:42.092888 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.094012 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.095177 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.095723 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.095856 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.098727 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:42.098733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:42.098748 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:42.098775 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:42.100861 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.101429 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.101434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.101438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.101556 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.101841 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.101843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.101844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.102359 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:42.102511 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.102830 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.103237 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":152,"threads_started":1}
I20260927 10:53:42.104825 23358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6116a552a6534fa286c6bc22898f756d. Latest consensus state: current_term: 1 leader_uuid: "6116a552a6534fa286c6bc22898f756d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6116a552a6534fa286c6bc22898f756d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33457 } } }
I20260927 10:53:42.105548 23358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:42.105436 23359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6116a552a6534fa286c6bc22898f756d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6116a552a6534fa286c6bc22898f756d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 33457 } } }
I20260927 10:53:42.105866 23359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:42.106637 23367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:42.109228 23367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:42.114708 23367 catalog_manager.cc:1347] Generated new cluster ID: da0cdd4a450140fab6f0618ea878ae61
I20260927 10:53:42.114781 23367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:42.123962 23367 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:42.125098 23367 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:42.130962 23367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d: Generated new TSK 0
I20260927 10:53:42.131688 23367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:53:42.163560 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33457
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:42.278330 23379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:42.278625 23379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:42.278669 23379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:42.282913 23379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:42.282995 23379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:42.283057 23379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:53:42.288805 23379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:42.289970 23379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:42.291047 23379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:42.300987 23384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:42.301054 23385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:42.301091 23379 server_base.cc:1034] running on GCE node
W20260927 10:53:42.301008 23387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:42.301626 23379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:42.302271 23379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:42.303462 23379 hybrid_clock.cc:648] HybridClock initialized: now 1790506422303442 us; error 29 us; skew 500 ppm
Sep 27 10:53:42 dist-test-slave-c60x krb5kdc[23272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:42.306342 23379 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:53:42.307308 23379 webserver.cc:460] Webserver started at http://127.6.31.129:38233/ using document root <none> and password file <none>
I20260927 10:53:42.307884 23379 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:42.307986 23379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:42.308211 23379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:42.310237 23379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "33c4932a7c2f47adb6919f8327a31447"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "854d2e182df7fb103ad4cea0ac93fc97"
server_key_iv: "38198f2ec18df71fa0d6587df7f2748d"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.310810 23379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "33c4932a7c2f47adb6919f8327a31447"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "854d2e182df7fb103ad4cea0ac93fc97"
server_key_iv: "38198f2ec18df71fa0d6587df7f2748d"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.314612 23379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:53:42.317265 23394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.318662 23379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:42.318835 23379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "33c4932a7c2f47adb6919f8327a31447"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "854d2e182df7fb103ad4cea0ac93fc97"
server_key_iv: "38198f2ec18df71fa0d6587df7f2748d"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.318959 23379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:42.340626 23379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:42.344266 23379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:42.344504 23379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:42.345151 23379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:42.346043 23379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:42.346122 23379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.346216 23379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:42.346267 23379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.355938 23379 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:43987
I20260927 10:53:42.355962 23507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:43987 every 8 connection(s)
I20260927 10:53:42.356945 23379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:42.360427 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 23379
I20260927 10:53:42.360639 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:42.361034 6270 external_mini_cluster.cc:1428] Setting key af67043207ddd13a10fee48a86b9d6bd
Sep 27 10:53:42 dist-test-slave-c60x krb5kdc[23272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:53:42.373970 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.358966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:39013 (local address 127.6.31.190:33457)
0927 10:53:42.359161 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.359166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.360278 (+ 1112us) server_negotiation.cc:407] Connection header received
0927 10:53:42.362049 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.362053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.362107 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.362187 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:42.363846 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.364503 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.366221 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.366662 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.366802 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.369963 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:42.369968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:42.369973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:42.370009 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:42.371787 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.372373 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.372376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.372377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.372454 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.372791 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.372794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.372795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.373032 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:42.373181 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.373720 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.373831 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:53:42.374992 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.359471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33457 (local address 127.6.31.129:39013)
0927 10:53:42.359960 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:42.360088 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:42.361774 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:42.362359 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:42.362366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:42.362857 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:42.363657 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.363680 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.364651 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.364656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.366052 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.366063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.366849 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.366853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.367032 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.367853 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:42.367871 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:42.369744 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:42.371948 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:42.371960 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:42.371975 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:42.372257 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:42.372539 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:42.372541 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:42.372542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:42.372689 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:42.373203 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:42.373209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:42.373469 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 10:53:42.373655 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 10:53:42.373936 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":129,"threads_started":1}
I20260927 10:53:42.376356 23508 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33457
I20260927 10:53:42.376657 23508 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:42.377202 23508 heartbeater.cc:511] Master 127.6.31.190:33457 requested a full tablet report, sending...
I20260927 10:53:42.378801 23317 ts_manager.cc:194] Registered new tserver with Master: 33c4932a7c2f47adb6919f8327a31447 (127.6.31.129:43987)
I20260927 10:53:42.380043 23317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:39013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:53:42.395340 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33457
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:42.507737 23513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:42.508021 23513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:42.508066 23513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:42.512285 23513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:42.512358 23513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:42.512420 23513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:53:42.517949 23513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:42.519135 23513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:42.520229 23513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:42.530273 23518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:42.530277 23519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:42.530273 23521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:42.530329 23513 server_base.cc:1034] running on GCE node
I20260927 10:53:42.530977 23513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:42.531709 23513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:42.532932 23513 hybrid_clock.cc:648] HybridClock initialized: now 1790506422532903 us; error 51 us; skew 500 ppm
Sep 27 10:53:42 dist-test-slave-c60x krb5kdc[23272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:42.535840 23513 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:53:42.536854 23513 webserver.cc:460] Webserver started at http://127.6.31.130:36485/ using document root <none> and password file <none>
I20260927 10:53:42.537438 23513 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:42.537600 23513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:42.537849 23513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:42.540007 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "a3465a907c8c4588bbb9000643fce21a"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "e75bcc1bec78dcbb8cd1d377e179f61d"
server_key_iv: "883344c440267a088b321ed506954246"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.540572 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "a3465a907c8c4588bbb9000643fce21a"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "e75bcc1bec78dcbb8cd1d377e179f61d"
server_key_iv: "883344c440267a088b321ed506954246"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.544550 23513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:53:42.547309 23528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.548630 23513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:42.548771 23513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "a3465a907c8c4588bbb9000643fce21a"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "e75bcc1bec78dcbb8cd1d377e179f61d"
server_key_iv: "883344c440267a088b321ed506954246"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.548884 23513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:42.577015 23513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:42.580562 23513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:42.580786 23513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:42.581403 23513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:42.582280 23513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:42.582326 23513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.582399 23513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:42.582441 23513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.591575 23513 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:43811
I20260927 10:53:42.591648 23641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:43811 every 8 connection(s)
I20260927 10:53:42.592521 23513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:53:42 dist-test-slave-c60x krb5kdc[23272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:42.602551 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 23513
I20260927 10:53:42.602696 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:42.603044 6270 external_mini_cluster.cc:1428] Setting key cd71e631c652f691a6fbf95dcb53dc37
I20260927 10:53:42.607236 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.594366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:43767 (local address 127.6.31.190:33457)
0927 10:53:42.594522 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.594527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.595553 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 10:53:42.596818 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.596820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.596866 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.596952 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:42.598200 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.598750 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.599953 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.600165 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.600244 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.602698 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:42.602704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:42.602710 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:42.602743 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:42.604473 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.605085 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.605088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.605090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.605196 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.605581 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.605586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.605587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.605780 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:42.605895 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.606920 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.607045 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:53:42.608166 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.594803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33457 (local address 127.6.31.130:43767)
0927 10:53:42.595399 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:42.595445 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:42.596623 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:42.597216 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:42.597226 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:42.597588 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:42.598058 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.598071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.598872 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.598875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.599846 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.599856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.600296 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.600299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.600439 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.601039 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:42.601056 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:42.602519 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:42.604601 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:42.604615 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:42.604634 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:42.604988 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:42.605331 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:42.605339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:42.605341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:42.605481 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:42.606282 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:42.606294 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:42.606638 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 10:53:42.606869 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 10:53:42.607248 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":113,"threads_started":1}
I20260927 10:53:42.610452 23642 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33457
I20260927 10:53:42.610741 23642 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:42.611431 23642 heartbeater.cc:511] Master 127.6.31.190:33457 requested a full tablet report, sending...
I20260927 10:53:42.612617 23317 ts_manager.cc:194] Registered new tserver with Master: a3465a907c8c4588bbb9000643fce21a (127.6.31.130:43811)
I20260927 10:53:42.613304 23317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:43767
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:53:42.634822 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33457
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:42.745035 23647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:42.745308 23647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:42.745354 23647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:42.749456 23647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:42.749524 23647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:42.749584 23647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:53:42.755193 23647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:33457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:42.756256 23647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:42.757273 23647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:42.766618 23655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:42.766659 23653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:42.766708 23647 server_base.cc:1034] running on GCE node
W20260927 10:53:42.766633 23652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:42.767343 23647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:42.767949 23647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:42.769155 23647 hybrid_clock.cc:648] HybridClock initialized: now 1790506422769136 us; error 29 us; skew 500 ppm
Sep 27 10:53:42 dist-test-slave-c60x krb5kdc[23272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:42.778745 23647 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:53:42.779728 23647 webserver.cc:460] Webserver started at http://127.6.31.131:39897/ using document root <none> and password file <none>
I20260927 10:53:42.780264 23647 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:42.780323 23647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:42.780536 23647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:42.782577 23647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "0fa073197710416a92da67407cf922b7"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "abc9c6a50e8d1071835502625c8ecf25"
server_key_iv: "a48e4e114862df0f650d8d44b63db19e"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.783131 23647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "0fa073197710416a92da67407cf922b7"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "abc9c6a50e8d1071835502625c8ecf25"
server_key_iv: "a48e4e114862df0f650d8d44b63db19e"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.786754 23647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:42.789119 23662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.790352 23647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:42.790490 23647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "0fa073197710416a92da67407cf922b7"
format_stamp: "Formatted at 2026-09-27 10:53:42 on dist-test-slave-c60x"
server_key: "abc9c6a50e8d1071835502625c8ecf25"
server_key_iv: "a48e4e114862df0f650d8d44b63db19e"
server_key_version: "encryptionkey@0"
I20260927 10:53:42.790611 23647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:42.802937 23647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:42.806123 23647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:42.806329 23647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:42.806932 23647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:42.807821 23647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:42.807868 23647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.807937 23647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:42.807976 23647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:42.816934 23647 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:33063
I20260927 10:53:42.816982 23775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:33063 every 8 connection(s)
I20260927 10:53:42.817871 23647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:42.821385 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 23647
I20260927 10:53:42.821519 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:42.821897 6270 external_mini_cluster.cc:1428] Setting key 81e3ec8f24a73a5ba97f284876a4e50f
Sep 27 10:53:42 dist-test-slave-c60x krb5kdc[23272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:42.833963 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.819812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:51919 (local address 127.6.31.190:33457)
0927 10:53:42.819982 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.819987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.821143 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 10:53:42.822854 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.822858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.822917 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.822988 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:42.824442 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.825104 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.826319 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.826562 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.826937 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.829878 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:42.829882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:42.829885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:42.829921 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:42.831659 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.832250 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.832253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.832256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.832363 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.832747 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.832751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.832753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.832964 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:42.833077 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.833653 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.833790 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:53:42.834975 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.820297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:33457 (local address 127.6.31.131:51919)
0927 10:53:42.820938 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:42.821011 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:42.822627 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:42.823216 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:42.823223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:42.823708 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:42.824298 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.824313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.825243 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.825246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.826180 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.826191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.827139 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.827147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.827334 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.828026 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:42.828044 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:42.829663 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:42.831801 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:42.831808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:42.831828 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:42.832130 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:42.832472 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:42.832475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:42.832477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:42.832638 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:42.833077 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:42.833085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:42.833386 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 10:53:42.833599 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 10:53:42.833980 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":181,"threads_started":1}
I20260927 10:53:42.836242 23776 heartbeater.cc:347] Connected to a master server at 127.6.31.190:33457
I20260927 10:53:42.836513 23776 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:42.837085 23776 heartbeater.cc:511] Master 127.6.31.190:33457 requested a full tablet report, sending...
I20260927 10:53:42.838132 23317 ts_manager.cc:194] Registered new tserver with Master: 0fa073197710416a92da67407cf922b7 (127.6.31.131:33063)
I20260927 10:53:42.838714 23317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:51919
I20260927 10:53:42.848500 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:42.858233 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.850584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.6.31.190:33457)
0927 10:53:42.850778 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.850782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.850820 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:53:42.850945 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.850947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.850990 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.851061 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:42.851773 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.852392 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.853290 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.853554 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.853636 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.855152 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:42.855157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:42.855164 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:42.855188 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:42.856424 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.856812 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.856815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.856816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.856974 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.857235 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.857238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.857239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.857471 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:42.857586 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.857821 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.858068 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 10:53:42.861245 23317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:42.864482 23317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:42.880043 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.874152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.6.31.130:43811)
0927 10:53:42.874682 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.874693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.874716 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:53:42.874799 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.874804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.875018 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.875395 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:42.875709 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.876598 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.878626 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.879285 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.879409 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.879569 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.879767 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.879865 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":150,"threads_started":1}
I20260927 10:53:42.880033 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.873400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:43811 (local address 127.0.0.1:57046)
0927 10:53:42.874157 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:42.874210 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:42.874360 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:42.875202 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:42.875208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:42.875237 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:42.875575 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.875584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.876832 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.876837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.878489 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.878500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.879429 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.879433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.879541 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.879577 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 10:53:42.879732 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 10:53:42.879867 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":329,"spinlock_wait_cycles":17408,"thread_start_us":99,"threads_started":1}
I20260927 10:53:42.880304 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.873322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:43987 (local address 127.0.0.1:58436)
0927 10:53:42.874156 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:42.874210 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:42.874359 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:42.875201 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:42.875209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:42.875241 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:42.875426 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.875437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.877132 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.877135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.878447 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.878459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.879784 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.879788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.880041 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.880059 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:53:42.880119 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 10:53:42.880169 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":97,"spinlock_wait_cycles":14720,"thread_start_us":76,"threads_started":1}
I20260927 10:53:42.880518 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.874014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.6.31.131:33063)
0927 10:53:42.874440 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.874454 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.874955 (+ 501us) server_negotiation.cc:407] Connection header received
0927 10:53:42.875208 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.875213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.875377 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.875521 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:42.875970 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.877075 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.878423 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.879456 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.879583 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.879716 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.880241 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.880338 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":96,"threads_started":1}
I20260927 10:53:42.880832 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.873473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.6.31.129:43987)
0927 10:53:42.874643 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.874655 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.874675 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:53:42.874763 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.874768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.874982 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.875725 (+ 743us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:42.875835 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.877008 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.878582 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.879508 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.879599 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.879716 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.880573 (+ 857us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.880680 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":146,"threads_started":1}
I20260927 10:53:42.881796 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.873680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:33063 (local address 127.0.0.1:51050)
0927 10:53:42.874864 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:42.874877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:42.875019 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:42.875734 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:42.875737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:42.875757 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:42.875833 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.875839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.877219 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.877228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.878302 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.878311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.879784 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.879788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.879934 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.879958 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:53:42.880034 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:53:42.880122 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":24,"thread_start_us":109,"threads_started":1}
I20260927 10:53:42.883778 23442 tablet_service.cc:1467] Processing CreateTablet for tablet d96946ec1592476eaeaf66cfcb2c0f18 (DEFAULT_TABLE table=test-table [id=cf918a9811514d5dacecc6b90bd92907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:42.883810 23576 tablet_service.cc:1467] Processing CreateTablet for tablet d96946ec1592476eaeaf66cfcb2c0f18 (DEFAULT_TABLE table=test-table [id=cf918a9811514d5dacecc6b90bd92907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:42.884341 23710 tablet_service.cc:1467] Processing CreateTablet for tablet d96946ec1592476eaeaf66cfcb2c0f18 (DEFAULT_TABLE table=test-table [id=cf918a9811514d5dacecc6b90bd92907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:42.884871 23576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96946ec1592476eaeaf66cfcb2c0f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:42.884889 23442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96946ec1592476eaeaf66cfcb2c0f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:42.885222 23710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96946ec1592476eaeaf66cfcb2c0f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:42.891216 23793 tablet_bootstrap.cc:492] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447: Bootstrap starting.
I20260927 10:53:42.892504 23794 tablet_bootstrap.cc:492] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: Bootstrap starting.
I20260927 10:53:42.893904 23793 tablet_bootstrap.cc:654] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:42.894589 23794 tablet_bootstrap.cc:654] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:42.894881 23793 log.cc:824] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:42.895393 23794 log.cc:824] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:42.895525 23795 tablet_bootstrap.cc:492] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: Bootstrap starting.
I20260927 10:53:42.897444 23794 tablet_bootstrap.cc:492] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: No bootstrap required, opened a new log
I20260927 10:53:42.897552 23793 tablet_bootstrap.cc:492] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447: No bootstrap required, opened a new log
I20260927 10:53:42.897670 23794 ts_tablet_manager.cc:1397] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:53:42.897742 23793 ts_tablet_manager.cc:1397] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 10:53:42.897746 23795 tablet_bootstrap.cc:654] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:42.898993 23795 log.cc:824] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:42.901126 23795 tablet_bootstrap.cc:492] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: No bootstrap required, opened a new log
I20260927 10:53:42.900993 23794 raft_consensus.cc:348] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } }
I20260927 10:53:42.901207 23794 raft_consensus.cc:374] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:42.901240 23794 raft_consensus.cc:729] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3465a907c8c4588bbb9000643fce21a, State: Initialized, Role: FOLLOWER
I20260927 10:53:42.901306 23795 ts_tablet_manager.cc:1397] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 10:53:42.901657 23794 consensus_queue.cc:260] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } }
I20260927 10:53:42.902253 23642 heartbeater.cc:503] Master 127.6.31.190:33457 was elected leader, sending a full tablet report...
I20260927 10:53:42.902618 23794 ts_tablet_manager.cc:1428] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:53:42.902582 23793 raft_consensus.cc:348] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } }
I20260927 10:53:42.902858 23793 raft_consensus.cc:374] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:42.902956 23793 raft_consensus.cc:729] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c4932a7c2f47adb6919f8327a31447, State: Initialized, Role: FOLLOWER
I20260927 10:53:42.903452 23793 consensus_queue.cc:260] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } }
I20260927 10:53:42.904309 23508 heartbeater.cc:503] Master 127.6.31.190:33457 was elected leader, sending a full tablet report...
I20260927 10:53:42.904523 23793 ts_tablet_manager.cc:1428] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260927 10:53:42.905323 23795 raft_consensus.cc:348] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } }
I20260927 10:53:42.905658 23795 raft_consensus.cc:374] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:42.905817 23795 raft_consensus.cc:729] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa073197710416a92da67407cf922b7, State: Initialized, Role: FOLLOWER
I20260927 10:53:42.906335 23795 consensus_queue.cc:260] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } }
I20260927 10:53:42.907042 23776 heartbeater.cc:503] Master 127.6.31.190:33457 was elected leader, sending a full tablet report...
I20260927 10:53:42.907364 23795 ts_tablet_manager.cc:1428] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
W20260927 10:53:43.068914 23777 tablet.cc:2369] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:43.093786 23643 tablet.cc:2369] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:43.096366 23799 raft_consensus.cc:482] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:43.096577 23799 raft_consensus.cc:504] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } }
I20260927 10:53:43.097791 23799 leader_election.cc:288] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c4932a7c2f47adb6919f8327a31447 (127.6.31.129:43987), 0fa073197710416a92da67407cf922b7 (127.6.31.131:33063)
I20260927 10:53:43.102801 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.098214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:58603 (local address 127.6.31.129:43987)
0927 10:53:43.098352 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:43.098357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:43.098375 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:53:43.098436 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:43.098438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:43.098509 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:43.098595 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:43.098922 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.099707 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.101428 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.102078 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.102210 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.102263 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:43.102591 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 10:53:43.102668 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:53:43.102801 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.098433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:41459 (local address 127.6.31.131:33063)
0927 10:53:43.098597 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:43.098601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:43.098637 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:53:43.098729 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:43.098732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:43.098771 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:43.098831 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:43.099194 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.099890 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.101428 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.102078 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.102212 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.102263 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:43.102591 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 10:53:43.102677 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:53:43.102843 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.098000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:43987 (local address 127.6.31.130:58603)
0927 10:53:43.098131 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.098149 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:43.098252 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:43.098630 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:43.098634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:43.098647 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:43.098786 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.098791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.099830 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.099832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.101298 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.101310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.102216 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.102220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.102384 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.102480 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 10:53:43.102562 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:53:43.102668 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":157}
I20260927 10:53:43.102851 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.098131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:33063 (local address 127.6.31.130:41459)
0927 10:53:43.098509 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.098525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:43.098641 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:43.098866 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:43.098872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:43.098893 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:43.099043 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.099048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.100043 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.100047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.101314 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.101320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.102216 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.102220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.102402 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.102475 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 10:53:43.102562 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:53:43.102668 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":155,"thread_start_us":108,"threads_started":1}
I20260927 10:53:43.103644 23462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96946ec1592476eaeaf66cfcb2c0f18" candidate_uuid: "a3465a907c8c4588bbb9000643fce21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c4932a7c2f47adb6919f8327a31447" is_pre_election: true
I20260927 10:53:43.103646 23730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96946ec1592476eaeaf66cfcb2c0f18" candidate_uuid: "a3465a907c8c4588bbb9000643fce21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa073197710416a92da67407cf922b7" is_pre_election: true
I20260927 10:53:43.103976 23730 raft_consensus.cc:2454] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3465a907c8c4588bbb9000643fce21a in term 0.
I20260927 10:53:43.103976 23462 raft_consensus.cc:2454] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3465a907c8c4588bbb9000643fce21a in term 0.
I20260927 10:53:43.104532 23532 leader_election.cc:302] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0fa073197710416a92da67407cf922b7, 33c4932a7c2f47adb6919f8327a31447, a3465a907c8c4588bbb9000643fce21a; no voters:
I20260927 10:53:43.104866 23799 raft_consensus.cc:2781] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:43.104949 23799 raft_consensus.cc:482] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:43.104980 23799 raft_consensus.cc:3037] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:43.106186 23799 raft_consensus.cc:504] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } }
I20260927 10:53:43.106554 23799 leader_election.cc:288] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [CANDIDATE]: Term 1 election: Requested vote from peers 33c4932a7c2f47adb6919f8327a31447 (127.6.31.129:43987), 0fa073197710416a92da67407cf922b7 (127.6.31.131:33063)
I20260927 10:53:43.106930 23462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96946ec1592476eaeaf66cfcb2c0f18" candidate_uuid: "a3465a907c8c4588bbb9000643fce21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c4932a7c2f47adb6919f8327a31447"
I20260927 10:53:43.107072 23462 raft_consensus.cc:3037] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447 [term 0 FOLLOWER]: Advancing to term 1
W20260927 10:53:43.108394 23509 tablet.cc:2369] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:43.108522 23462 raft_consensus.cc:2454] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3465a907c8c4588bbb9000643fce21a in term 1.
I20260927 10:53:43.108807 23730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96946ec1592476eaeaf66cfcb2c0f18" candidate_uuid: "a3465a907c8c4588bbb9000643fce21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa073197710416a92da67407cf922b7"
I20260927 10:53:43.108891 23531 leader_election.cc:302] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c4932a7c2f47adb6919f8327a31447, a3465a907c8c4588bbb9000643fce21a; no voters:
I20260927 10:53:43.108969 23730 raft_consensus.cc:3037] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:43.109058 23799 raft_consensus.cc:2781] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:43.109308 23799 raft_consensus.cc:686] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 1 LEADER]: Becoming Leader. State: Replica: a3465a907c8c4588bbb9000643fce21a, State: Running, Role: LEADER
I20260927 10:53:43.109651 23799 consensus_queue.cc:237] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } }
I20260927 10:53:43.110095 23730 raft_consensus.cc:2454] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3465a907c8c4588bbb9000643fce21a in term 1.
I20260927 10:53:43.113058 23317 catalog_manager.cc:5495] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a reported cstate change: term changed from 0 to 1, leader changed from <none> to a3465a907c8c4588bbb9000643fce21a (127.6.31.130). New cstate: current_term: 1 leader_uuid: "a3465a907c8c4588bbb9000643fce21a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3465a907c8c4588bbb9000643fce21a" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 43811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:43.173321 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.169608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.6.31.130:43811)
0927 10:53:43.169753 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:43.169756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:43.169771 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:43.169849 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:43.169851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:43.169906 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:43.169998 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:43.170310 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.170813 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.171762 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.172044 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.172138 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.172424 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:43.172860 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:43.173010 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:43.173038 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 10:53:43.173095 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:53:43.179561 23462 raft_consensus.cc:1261] T d96946ec1592476eaeaf66cfcb2c0f18 P 33c4932a7c2f47adb6919f8327a31447 [term 1 FOLLOWER]: Refusing update from remote peer a3465a907c8c4588bbb9000643fce21a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:43.179561 23730 raft_consensus.cc:1261] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [term 1 FOLLOWER]: Refusing update from remote peer a3465a907c8c4588bbb9000643fce21a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:43.180281 23799 consensus_queue.cc:1035] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c4932a7c2f47adb6919f8327a31447" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:43.180500 23803 consensus_queue.cc:1035] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa073197710416a92da67407cf922b7" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:43.188423 23807 mvcc.cc:205] Tried to move back new op lower bound from 7333914309335429120 to 7333914309060341760. Current Snapshot: MvccSnapshot[applied={T|T < 7333914309335429120}]
I20260927 10:53:43.190058 23808 mvcc.cc:205] Tried to move back new op lower bound from 7333914309335429120 to 7333914309060341760. Current Snapshot: MvccSnapshot[applied={T|T < 7333914309335429120}]
I20260927 10:53:43.197389 23317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:43.197654 23317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:43.200654 23317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6116a552a6534fa286c6bc22898f756d: Sending DeleteTablet for 3 replicas of tablet d96946ec1592476eaeaf66cfcb2c0f18
I20260927 10:53:43.201527 23442 tablet_service.cc:1514] Processing DeleteTablet for tablet d96946ec1592476eaeaf66cfcb2c0f18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:58436
I20260927 10:53:43.201542 23576 tablet_service.cc:1514] Processing DeleteTablet for tablet d96946ec1592476eaeaf66cfcb2c0f18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:57046
I20260927 10:53:43.201542 23710 tablet_service.cc:1514] Processing DeleteTablet for tablet d96946ec1592476eaeaf66cfcb2c0f18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:43 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:51050
I20260927 10:53:43.202099 23817 tablet_replica.cc:330] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: stopping tablet replica
I20260927 10:53:43.202436 23817 raft_consensus.cc:2229] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:43.202797 23817 raft_consensus.cc:2258] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:43.202999 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 23379
I20260927 10:53:43.203236 23818 tablet_replica.cc:330] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: stopping tablet replica
I20260927 10:53:43.203545 23818 raft_consensus.cc:2229] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:43.203898 23818 raft_consensus.cc:2258] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:43.204727 23817 ts_tablet_manager.cc:1905] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:43.205224 23818 ts_tablet_manager.cc:1905] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:43.207764 23817 ts_tablet_manager.cc:1918] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:43.207909 23817 log.cc:1196] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/d96946ec1592476eaeaf66cfcb2c0f18
I20260927 10:53:43.208115 23818 ts_tablet_manager.cc:1918] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:43.208235 23818 log.cc:1196] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/d96946ec1592476eaeaf66cfcb2c0f18
I20260927 10:53:43.208326 23817 ts_tablet_manager.cc:1939] T d96946ec1592476eaeaf66cfcb2c0f18 P 0fa073197710416a92da67407cf922b7: Deleting consensus metadata
I20260927 10:53:43.208660 23818 ts_tablet_manager.cc:1939] T d96946ec1592476eaeaf66cfcb2c0f18 P a3465a907c8c4588bbb9000643fce21a: Deleting consensus metadata
I20260927 10:53:43.209738 23301 catalog_manager.cc:4841] TS a3465a907c8c4588bbb9000643fce21a (127.6.31.130:43811): tablet d96946ec1592476eaeaf66cfcb2c0f18 (table test-table [id=cf918a9811514d5dacecc6b90bd92907]) successfully deleted
I20260927 10:53:43.209745 23304 catalog_manager.cc:4841] TS 0fa073197710416a92da67407cf922b7 (127.6.31.131:33063): tablet d96946ec1592476eaeaf66cfcb2c0f18 (table test-table [id=cf918a9811514d5dacecc6b90bd92907]) successfully deleted
W20260927 10:53:43.214274 23303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:43987 (error 108)
I20260927 10:53:43.214505 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 23513
I20260927 10:53:43.215958 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.215549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:43987 (local address 127.0.0.1:58448)
0927 10:53:43.215796 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.215868 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260927 10:53:43.216518 23303 catalog_manager.cc:4584] TS 33c4932a7c2f47adb6919f8327a31447 (127.6.31.129:43987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d96946ec1592476eaeaf66cfcb2c0f18: Network error: Client connection negotiation failed: client connection to 127.6.31.129:43987: connect: Connection refused (error 111)
I20260927 10:53:43.220577 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 23647
I20260927 10:53:43.226267 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 23285
2026-09-27T10:53:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2507 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T10:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:43Z Disabled control of system clock
I20260927 10:53:43.257264 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37387
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:53:43.371495 23823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:43.371768 23823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:43.371802 23823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:43.375891 23823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:43.375963 23823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:43.375979 23823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:43.375998 23823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:43.376037 23823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:43.380959 23823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:37387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:37387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:43.382089 23823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:43.383502 23823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:43.392053 23829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:43.392048 23828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:43.392175 23823 server_base.cc:1034] running on GCE node
W20260927 10:53:43.392097 23831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:43.392913 23823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:43.394032 23823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:43.395187 23823 hybrid_clock.cc:648] HybridClock initialized: now 1790506423395158 us; error 40 us; skew 500 ppm
I20260927 10:53:43.397003 23823 webserver.cc:460] Webserver started at http://127.6.31.190:38867/ using document root <none> and password file <none>
I20260927 10:53:43.397614 23823 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:43.397707 23823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:43.397933 23823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:43.400116 23823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "8743a7442f69402b8ebcb4aa70188f92"
format_stamp: "Formatted at 2026-09-27 10:53:43 on dist-test-slave-c60x"
server_key: "a365fcd253b31bb6e10116a522233fee"
server_key_iv: "a16bb573220b3759dd9f21f9a2ec01d6"
server_key_version: "encryptionkey@0"
I20260927 10:53:43.400701 23823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "8743a7442f69402b8ebcb4aa70188f92"
format_stamp: "Formatted at 2026-09-27 10:53:43 on dist-test-slave-c60x"
server_key: "a365fcd253b31bb6e10116a522233fee"
server_key_iv: "a16bb573220b3759dd9f21f9a2ec01d6"
server_key_version: "encryptionkey@0"
I20260927 10:53:43.404484 23823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:53:43.406927 23837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:43.408238 23823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:43.408383 23823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "8743a7442f69402b8ebcb4aa70188f92"
format_stamp: "Formatted at 2026-09-27 10:53:43 on dist-test-slave-c60x"
server_key: "a365fcd253b31bb6e10116a522233fee"
server_key_iv: "a16bb573220b3759dd9f21f9a2ec01d6"
server_key_version: "encryptionkey@0"
I20260927 10:53:43.408496 23823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:43.425223 23823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:43.425938 23823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:43.426157 23823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:43.433442 23823 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:37387
I20260927 10:53:43.433467 23889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:37387 every 8 connection(s)
I20260927 10:53:43.434409 23823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:43.437587 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:43.443403 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 23823
I20260927 10:53:43.443540 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:43.443828 6270 external_mini_cluster.cc:1428] Setting key 894fd6f87999319ccb2b3c8f080915c4
I20260927 10:53:43.443881 23890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92: Bootstrap starting.
I20260927 10:53:43.446507 23890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:43.447329 23890 log.cc:824] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:43.449186 23890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92: No bootstrap required, opened a new log
I20260927 10:53:43.452987 23890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8743a7442f69402b8ebcb4aa70188f92" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37387 } }
I20260927 10:53:43.453262 23890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:43.453344 23890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8743a7442f69402b8ebcb4aa70188f92, State: Initialized, Role: FOLLOWER
I20260927 10:53:43.453622 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.445344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.6.31.190:37387)
0927 10:53:43.445980 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:43.445994 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:43.446057 (+ 63us) server_negotiation.cc:407] Connection header received
0927 10:53:43.446778 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:43.446796 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:43.447264 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:43.447637 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:43.448109 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.449103 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.450078 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.450564 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.450734 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.451688 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:43.451699 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:43.451712 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:43.452114 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:43.452169 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:43.452187 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:43.452301 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:43.452456 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 10:53:43.452726 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260927 10:53:43.453856 23890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8743a7442f69402b8ebcb4aa70188f92" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37387 } }
I20260927 10:53:43.454017 23890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:43.454075 23890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:43.454167 23890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:43.455265 23890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8743a7442f69402b8ebcb4aa70188f92" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37387 } }
I20260927 10:53:43.455600 23890 leader_election.cc:302] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8743a7442f69402b8ebcb4aa70188f92; no voters:
I20260927 10:53:43.455957 23890 leader_election.cc:288] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:43.456086 23895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:43.456346 23895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [term 1 LEADER]: Becoming Leader. State: Replica: 8743a7442f69402b8ebcb4aa70188f92, State: Running, Role: LEADER
I20260927 10:53:43.456691 23895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8743a7442f69402b8ebcb4aa70188f92" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37387 } }
I20260927 10:53:43.457151 23890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:43.458424 23897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8743a7442f69402b8ebcb4aa70188f92. Latest consensus state: current_term: 1 leader_uuid: "8743a7442f69402b8ebcb4aa70188f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8743a7442f69402b8ebcb4aa70188f92" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37387 } } }
I20260927 10:53:43.458547 23897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:43.459013 23896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8743a7442f69402b8ebcb4aa70188f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8743a7442f69402b8ebcb4aa70188f92" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 37387 } } }
I20260927 10:53:43.459152 23896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:43.459556 23904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:43.462217 23904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:43.467444 23904 catalog_manager.cc:1347] Generated new cluster ID: 8daeca254b8140d5a682441d7a6de09b
I20260927 10:53:43.467548 23904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:43.499262 23904 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:43.500223 23904 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:43.514881 23904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92: Generated new TSK 0
I20260927 10:53:43.515663 23904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:43.527822 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37387
--builtin_ntp_servers=127.6.31.148:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:53:43.638863 23914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:43.639220 23914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:43.639299 23914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:43.643399 23914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:43.643489 23914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:43.643584 23914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:53:43.648392 23914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:43.649514 23914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:43.650879 23914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:43.660893 23920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:43.660893 23922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:43.660938 23919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:43.661090 23914 server_base.cc:1034] running on GCE node
I20260927 10:53:43.661584 23914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:43.662195 23914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:43.663352 23914 hybrid_clock.cc:648] HybridClock initialized: now 1790506423663338 us; error 25 us; skew 500 ppm
I20260927 10:53:43.665227 23914 webserver.cc:460] Webserver started at http://127.6.31.129:35805/ using document root <none> and password file <none>
I20260927 10:53:43.665807 23914 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:43.665868 23914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:43.666096 23914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:43.668139 23914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "123f4fb45eff4c3c93f002a2c52ae4f1"
format_stamp: "Formatted at 2026-09-27 10:53:43 on dist-test-slave-c60x"
server_key: "8c90d07ea1c77b7e46494eb23f99f51e"
server_key_iv: "f7ceb9612d30bf4442eca0521db2de94"
server_key_version: "encryptionkey@0"
I20260927 10:53:43.668684 23914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "123f4fb45eff4c3c93f002a2c52ae4f1"
format_stamp: "Formatted at 2026-09-27 10:53:43 on dist-test-slave-c60x"
server_key: "8c90d07ea1c77b7e46494eb23f99f51e"
server_key_iv: "f7ceb9612d30bf4442eca0521db2de94"
server_key_version: "encryptionkey@0"
I20260927 10:53:43.672281 23914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:43.674576 23928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:43.675820 23914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:43.675951 23914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "123f4fb45eff4c3c93f002a2c52ae4f1"
format_stamp: "Formatted at 2026-09-27 10:53:43 on dist-test-slave-c60x"
server_key: "8c90d07ea1c77b7e46494eb23f99f51e"
server_key_iv: "f7ceb9612d30bf4442eca0521db2de94"
server_key_version: "encryptionkey@0"
I20260927 10:53:43.676064 23914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:43.697592 23914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:43.698385 23914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:43.698594 23914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:43.699272 23914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:43.700197 23914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:43.700269 23914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:43.700337 23914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:43.700387 23914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:43.709532 23914 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:35233
I20260927 10:53:43.709558 24041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:35233 every 8 connection(s)
I20260927 10:53:43.710441 23914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:43.714675 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 23914
I20260927 10:53:43.714766 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:43.715080 6270 external_mini_cluster.cc:1428] Setting key a6bafa548bed51546c63649815b3df34
I20260927 10:53:43.717103 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37387
--builtin_ntp_servers=127.6.31.148:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:53:43.720315 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.712374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:32901 (local address 127.6.31.190:37387)
0927 10:53:43.712508 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:43.712512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:43.713580 (+ 1068us) server_negotiation.cc:407] Connection header received
0927 10:53:43.715043 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:43.715047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:43.715138 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:43.715245 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:43.716207 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.716854 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.718089 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.718485 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.718603 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.719744 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:43.719751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:43.719757 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:43.719774 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:43.719799 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:43.719812 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:43.719893 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:43.720037 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 10:53:43.720105 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:53:43.721062 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.712799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37387 (local address 127.6.31.129:32901)
0927 10:53:43.713446 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.713488 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:43.714805 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:43.715387 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:43.715393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:43.715472 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:43.716059 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.716073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.716997 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.717000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.717943 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.717952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.718644 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.718647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.718793 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.719471 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:43.719473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:43.719512 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:43.719581 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:43.719899 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:43.719905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:43.719909 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:53:43.720000 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:53:43.720213 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260927 10:53:43.722301 24042 heartbeater.cc:347] Connected to a master server at 127.6.31.190:37387
I20260927 10:53:43.722515 24042 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:43.723013 24042 heartbeater.cc:511] Master 127.6.31.190:37387 requested a full tablet report, sending...
I20260927 10:53:43.724510 23854 ts_manager.cc:194] Registered new tserver with Master: 123f4fb45eff4c3c93f002a2c52ae4f1 (127.6.31.129:35233)
I20260927 10:53:43.725634 23854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.129:32901
W20260927 10:53:43.832438 24045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:43.832811 24045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:43.832890 24045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:43.837107 24045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:43.837219 24045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:43.837317 24045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:53:43.842459 24045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:43.843672 24045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:43.845080 24045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:43.854816 24051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:43.854846 24053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:43.854807 24050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:43.854827 24045 server_base.cc:1034] running on GCE node
I20260927 10:53:43.855468 24045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:43.856184 24045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:43.857383 24045 hybrid_clock.cc:648] HybridClock initialized: now 1790506423857361 us; error 39 us; skew 500 ppm
I20260927 10:53:43.859557 24045 webserver.cc:460] Webserver started at http://127.6.31.130:35869/ using document root <none> and password file <none>
I20260927 10:53:43.860159 24045 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:43.860253 24045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:43.860484 24045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:43.862586 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "1d7717733275464198449a77396588d6"
format_stamp: "Formatted at 2026-09-27 10:53:43 on dist-test-slave-c60x"
server_key: "2483dcb44a521fe04fab5703deb26d7c"
server_key_iv: "9185f416d029441f492e7bb8de59efda"
server_key_version: "encryptionkey@0"
I20260927 10:53:43.863253 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "1d7717733275464198449a77396588d6"
format_stamp: "Formatted at 2026-09-27 10:53:43 on dist-test-slave-c60x"
server_key: "2483dcb44a521fe04fab5703deb26d7c"
server_key_iv: "9185f416d029441f492e7bb8de59efda"
server_key_version: "encryptionkey@0"
I20260927 10:53:43.866948 24045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:43.869405 24059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:43.870580 24045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:43.870726 24045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "1d7717733275464198449a77396588d6"
format_stamp: "Formatted at 2026-09-27 10:53:43 on dist-test-slave-c60x"
server_key: "2483dcb44a521fe04fab5703deb26d7c"
server_key_iv: "9185f416d029441f492e7bb8de59efda"
server_key_version: "encryptionkey@0"
I20260927 10:53:43.870847 24045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:43.900848 24045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:43.901588 24045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:43.901809 24045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:43.902423 24045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:43.903431 24045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:43.903640 24045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:43.903760 24045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:43.903820 24045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:43.913036 24045 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:44313
I20260927 10:53:43.913110 24172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:44313 every 8 connection(s)
I20260927 10:53:43.914057 24045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:43.923456 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.915971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:37683 (local address 127.6.31.190:37387)
0927 10:53:43.916156 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:43.916159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:43.917190 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 10:53:43.918596 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:43.918600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:43.918663 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:43.918767 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:43.919747 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.920292 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.921433 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.921666 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.921783 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.922856 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:43.922859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:43.922862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:43.922881 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:43.922899 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:43.922907 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:43.922983 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:43.923243 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 10:53:43.923306 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 10:53:43.924386 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 24045
I20260927 10:53:43.924458 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.916309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37387 (local address 127.6.31.130:37683)
0927 10:53:43.916997 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.917073 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:43.918362 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:43.918943 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:43.918950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:43.919034 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:43.919603 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.919615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.920426 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.920430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.921302 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.921311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.921825 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.921829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.921996 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.922597 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:43.922599 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:43.922638 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:43.922700 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:43.922997 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:43.923007 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:43.923013 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:53:43.923244 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 10:53:43.923521 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260927 10:53:43.924535 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:43.924955 6270 external_mini_cluster.cc:1428] Setting key 0ea9f69e607835ca65817d29f4984756
I20260927 10:53:43.925709 24173 heartbeater.cc:347] Connected to a master server at 127.6.31.190:37387
I20260927 10:53:43.926041 24173 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:43.926607 24173 heartbeater.cc:511] Master 127.6.31.190:37387 requested a full tablet report, sending...
I20260927 10:53:43.927131 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37387
--builtin_ntp_servers=127.6.31.148:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:53:43.927744 23854 ts_manager.cc:194] Registered new tserver with Master: 1d7717733275464198449a77396588d6 (127.6.31.130:44313)
I20260927 10:53:43.928560 23854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.130:37683
W20260927 10:53:44.043913 24176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:44.044157 24176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:44.044219 24176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:44.048243 24176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:44.048300 24176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:44.048382 24176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:53:44.053277 24176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:44.054271 24176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:44.055675 24176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:44.065176 24184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:44.065196 24181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:44.065197 24182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:44.065589 24176 server_base.cc:1034] running on GCE node
I20260927 10:53:44.066025 24176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:44.066634 24176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:44.067814 24176 hybrid_clock.cc:648] HybridClock initialized: now 1790506424067782 us; error 44 us; skew 500 ppm
I20260927 10:53:44.069696 24176 webserver.cc:460] Webserver started at http://127.6.31.131:41957/ using document root <none> and password file <none>
I20260927 10:53:44.070299 24176 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:44.070396 24176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:44.070628 24176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:44.072772 24176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "8ebb03ff175b4b1ab2a0df5097c84879"
format_stamp: "Formatted at 2026-09-27 10:53:44 on dist-test-slave-c60x"
server_key: "0cd79291cdd8ca75a03a33b07e7ccbdc"
server_key_iv: "1b7e9e63aab48a9931cd240387fa652e"
server_key_version: "encryptionkey@0"
I20260927 10:53:44.073320 24176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "8ebb03ff175b4b1ab2a0df5097c84879"
format_stamp: "Formatted at 2026-09-27 10:53:44 on dist-test-slave-c60x"
server_key: "0cd79291cdd8ca75a03a33b07e7ccbdc"
server_key_iv: "1b7e9e63aab48a9931cd240387fa652e"
server_key_version: "encryptionkey@0"
I20260927 10:53:44.077035 24176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:44.079581 24190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:44.080869 24176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:44.080987 24176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "8ebb03ff175b4b1ab2a0df5097c84879"
format_stamp: "Formatted at 2026-09-27 10:53:44 on dist-test-slave-c60x"
server_key: "0cd79291cdd8ca75a03a33b07e7ccbdc"
server_key_iv: "1b7e9e63aab48a9931cd240387fa652e"
server_key_version: "encryptionkey@0"
I20260927 10:53:44.081102 24176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:44.092777 24176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:44.093400 24176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:44.093587 24176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:44.094175 24176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:44.095139 24176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:44.095213 24176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:44.095281 24176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:44.095331 24176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:44.104059 24176 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:38177
I20260927 10:53:44.104087 24303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:38177 every 8 connection(s)
I20260927 10:53:44.104907 24176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:44.113574 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 24176
I20260927 10:53:44.113701 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:44.113904 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.106740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:48187 (local address 127.6.31.190:37387)
0927 10:53:44.106885 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:44.106888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:44.107913 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 10:53:44.109193 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:44.109195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:44.109241 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:44.109314 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:44.110157 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.110678 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.111921 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.112149 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.112233 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.113423 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:44.113426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:44.113428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:44.113439 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:44.113457 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:44.113465 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:44.113523 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:44.113730 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 10:53:44.113794 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:53:44.113999 6270 external_mini_cluster.cc:1428] Setting key 26fdb8bbe7f2e05f8a10199a5456e1f6
I20260927 10:53:44.115899 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.107165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:37387 (local address 127.6.31.131:48187)
0927 10:53:44.107775 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.107811 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:44.109009 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:44.109463 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:44.109469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:44.109545 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:44.109994 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.110005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.110781 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.110784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.111738 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.111796 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.112277 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.112280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.112459 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.113150 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:44.113151 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:44.113188 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:44.113244 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:44.113552 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:44.113564 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:44.113569 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:53:44.113703 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 10:53:44.113973 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260927 10:53:44.117057 24304 heartbeater.cc:347] Connected to a master server at 127.6.31.190:37387
I20260927 10:53:44.117278 24304 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:44.117762 24304 heartbeater.cc:511] Master 127.6.31.190:37387 requested a full tablet report, sending...
I20260927 10:53:44.118707 23854 ts_manager.cc:194] Registered new tserver with Master: 8ebb03ff175b4b1ab2a0df5097c84879 (127.6.31.131:38177)
I20260927 10:53:44.119329 23854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.131:48187
I20260927 10:53:44.128093 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:44.133595 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.130011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.6.31.190:37387)
0927 10:53:44.130156 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:44.130160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:44.130375 (+ 215us) server_negotiation.cc:407] Connection header received
0927 10:53:44.130475 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:44.130477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:44.130517 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:44.130586 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:44.130823 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.131416 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.132241 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.132440 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.132514 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.133245 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:44.133248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:44.133250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:44.133262 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:44.133279 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:44.133287 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:44.133343 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:44.133436 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 10:53:44.133485 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:44.136332 23854 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:44.139412 23854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:44.153687 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.147574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:35233 (local address 127.0.0.1:47414)
0927 10:53:44.148363 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.148378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:44.148454 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:44.149451 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:44.149454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:44.149470 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:44.149555 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.149563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.150591 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.150598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.152232 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.152243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.153144 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.153147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.153250 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.153295 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 10:53:44.153415 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:53:44.153540 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":64,"thread_start_us":99,"threads_started":1}
I20260927 10:53:44.153703 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.148182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.6.31.129:35233)
0927 10:53:44.148986 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:44.148994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:44.149013 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:53:44.149071 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:44.149075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:44.149322 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:44.149466 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:44.149682 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.150465 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.152362 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.153026 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.153144 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.153334 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:44.153440 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 10:53:44.153529 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":104,"threads_started":1}
I20260927 10:53:44.154508 24316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.148025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38177 (local address 127.0.0.1:60914)
0927 10:53:44.148670 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.148682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:44.148762 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:44.149348 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:44.149351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:44.149373 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:44.149493 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.149499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.150945 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.150948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.152489 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.152499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.153976 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.153981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.154139 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.154204 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 10:53:44.154285 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:53:44.154366 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":111,"thread_start_us":106,"threads_started":1}
I20260927 10:53:44.154556 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.148442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.6.31.131:38177)
0927 10:53:44.148867 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:44.148877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:44.148895 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:53:44.148955 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:44.148959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:44.149111 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:44.149226 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:44.149628 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.150826 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.152616 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.153443 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.153546 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.153718 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:44.154314 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 10:53:44.154406 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":152,"threads_started":1}
I20260927 10:53:44.155205 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.147378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:44313 (local address 127.0.0.1:41636)
0927 10:53:44.147935 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.147984 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:44.148119 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:44.149184 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:44.149190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:44.149213 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:44.149343 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.149350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.150945 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.150948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.152934 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.152943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.153979 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.153982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.154143 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.154204 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 10:53:44.154983 (+ 779us) client_negotiation.cc:240] Negotiation successful
0927 10:53:44.155064 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":954,"thread_start_us":86,"threads_started":1}
I20260927 10:53:44.155397 24317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.147567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.6.31.130:44313)
0927 10:53:44.148695 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:44.148709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:44.148737 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:53:44.148813 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:44.148818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:44.149014 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:44.149181 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:44.149633 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.150814 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.153074 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.153835 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.154036 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.154213 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:44.155022 (+ 809us) server_negotiation.cc:299] Negotiation successful
0927 10:53:44.155138 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":124,"threads_started":1}
I20260927 10:53:44.156973 23976 tablet_service.cc:1467] Processing CreateTablet for tablet 52a5adbccf904e668eb6441ba0a6d147 (DEFAULT_TABLE table=test-table [id=970d75df45aa4e06aa2d4f138efaf57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:44.157446 24238 tablet_service.cc:1467] Processing CreateTablet for tablet 52a5adbccf904e668eb6441ba0a6d147 (DEFAULT_TABLE table=test-table [id=970d75df45aa4e06aa2d4f138efaf57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:44.158138 23976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a5adbccf904e668eb6441ba0a6d147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:44.158053 24107 tablet_service.cc:1467] Processing CreateTablet for tablet 52a5adbccf904e668eb6441ba0a6d147 (DEFAULT_TABLE table=test-table [id=970d75df45aa4e06aa2d4f138efaf57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:44.158456 24238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a5adbccf904e668eb6441ba0a6d147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:44.158979 24107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a5adbccf904e668eb6441ba0a6d147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:44.164206 24321 tablet_bootstrap.cc:492] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1: Bootstrap starting.
I20260927 10:53:44.165772 24323 tablet_bootstrap.cc:492] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6: Bootstrap starting.
I20260927 10:53:44.166488 24321 tablet_bootstrap.cc:654] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:44.167305 24321 log.cc:824] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:44.167420 24322 tablet_bootstrap.cc:492] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: Bootstrap starting.
I20260927 10:53:44.167749 24323 tablet_bootstrap.cc:654] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:44.168552 24323 log.cc:824] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:44.169507 24322 tablet_bootstrap.cc:654] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:44.169793 24321 tablet_bootstrap.cc:492] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1: No bootstrap required, opened a new log
I20260927 10:53:44.169988 24321 ts_tablet_manager.cc:1397] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:53:44.170244 24322 log.cc:824] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:44.171895 24322 tablet_bootstrap.cc:492] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: No bootstrap required, opened a new log
I20260927 10:53:44.172086 24322 ts_tablet_manager.cc:1397] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:53:44.173311 24321 raft_consensus.cc:348] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } }
I20260927 10:53:44.173552 24321 raft_consensus.cc:374] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:44.173640 24321 raft_consensus.cc:729] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123f4fb45eff4c3c93f002a2c52ae4f1, State: Initialized, Role: FOLLOWER
I20260927 10:53:44.174144 24321 consensus_queue.cc:260] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } }
I20260927 10:53:44.174844 24042 heartbeater.cc:503] Master 127.6.31.190:37387 was elected leader, sending a full tablet report...
I20260927 10:53:44.175223 24321 ts_tablet_manager.cc:1428] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260927 10:53:44.175743 24322 raft_consensus.cc:348] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } }
I20260927 10:53:44.176019 24322 raft_consensus.cc:374] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:44.176115 24322 raft_consensus.cc:729] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebb03ff175b4b1ab2a0df5097c84879, State: Initialized, Role: FOLLOWER
I20260927 10:53:44.176577 24322 consensus_queue.cc:260] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } }
I20260927 10:53:44.177176 24323 tablet_bootstrap.cc:492] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6: No bootstrap required, opened a new log
I20260927 10:53:44.177238 24304 heartbeater.cc:503] Master 127.6.31.190:37387 was elected leader, sending a full tablet report...
I20260927 10:53:44.177373 24323 ts_tablet_manager.cc:1397] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260927 10:53:44.177587 24322 ts_tablet_manager.cc:1428] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:53:44.181880 24323 raft_consensus.cc:348] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } }
I20260927 10:53:44.182149 24323 raft_consensus.cc:374] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:44.182219 24323 raft_consensus.cc:729] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7717733275464198449a77396588d6, State: Initialized, Role: FOLLOWER
I20260927 10:53:44.182665 24323 consensus_queue.cc:260] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } }
I20260927 10:53:44.183415 24173 heartbeater.cc:503] Master 127.6.31.190:37387 was elected leader, sending a full tablet report...
I20260927 10:53:44.183656 24323 ts_tablet_manager.cc:1428] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 10:53:44.211747 24043 tablet.cc:2369] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:44.242976 24329 raft_consensus.cc:482] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:44.243258 24329 raft_consensus.cc:504] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } }
I20260927 10:53:44.244504 24329 leader_election.cc:288] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123f4fb45eff4c3c93f002a2c52ae4f1 (127.6.31.129:35233), 8ebb03ff175b4b1ab2a0df5097c84879 (127.6.31.131:38177)
I20260927 10:53:44.249313 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.244782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:59101 (local address 127.6.31.129:35233)
0927 10:53:44.244930 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:44.244937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:44.244981 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:53:44.245050 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:44.245053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:44.245108 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:44.245168 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:44.245502 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.246319 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.248012 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.248618 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.248749 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.248791 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:44.249135 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 10:53:44.249196 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:53:44.249313 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.244960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:49909 (local address 127.6.31.131:38177)
0927 10:53:44.245075 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:44.245080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:44.245300 (+ 220us) server_negotiation.cc:407] Connection header received
0927 10:53:44.245375 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:44.245377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:44.245415 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:44.245509 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:44.245739 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.246438 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.248012 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.248618 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.248742 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.248791 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:44.249114 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 10:53:44.249183 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:53:44.249325 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.244677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:35233 (local address 127.6.31.130:59101)
0927 10:53:44.244814 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.244830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:44.244919 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:44.245209 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:44.245213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:44.245238 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:44.245373 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.245382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.246438 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.246440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.247882 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.247891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.248742 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.248746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.248917 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.248986 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 10:53:44.249067 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:53:44.249158 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":130}
I20260927 10:53:44.249332 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.244773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:38177 (local address 127.6.31.130:49909)
0927 10:53:44.245196 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.245209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:44.245295 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:44.245538 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:44.245540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:44.245553 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:44.245639 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.245643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.246538 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.246541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.247880 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:44.247891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.248742 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.248746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:44.248941 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.248994 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 10:53:44.249067 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:53:44.249158 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":161,"thread_start_us":137,"threads_started":1}
I20260927 10:53:44.250100 23996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a5adbccf904e668eb6441ba0a6d147" candidate_uuid: "1d7717733275464198449a77396588d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" is_pre_election: true
I20260927 10:53:44.250167 24258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a5adbccf904e668eb6441ba0a6d147" candidate_uuid: "1d7717733275464198449a77396588d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" is_pre_election: true
I20260927 10:53:44.250433 24258 raft_consensus.cc:2454] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d7717733275464198449a77396588d6 in term 0.
I20260927 10:53:44.250433 23996 raft_consensus.cc:2454] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d7717733275464198449a77396588d6 in term 0.
I20260927 10:53:44.250914 24063 leader_election.cc:302] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123f4fb45eff4c3c93f002a2c52ae4f1, 1d7717733275464198449a77396588d6; no voters:
I20260927 10:53:44.251199 24329 raft_consensus.cc:2781] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:44.251296 24329 raft_consensus.cc:482] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:44.251341 24329 raft_consensus.cc:3037] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:44.252368 24329 raft_consensus.cc:504] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } }
I20260927 10:53:44.252738 24329 leader_election.cc:288] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [CANDIDATE]: Term 1 election: Requested vote from peers 123f4fb45eff4c3c93f002a2c52ae4f1 (127.6.31.129:35233), 8ebb03ff175b4b1ab2a0df5097c84879 (127.6.31.131:38177)
I20260927 10:53:44.253024 23996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a5adbccf904e668eb6441ba0a6d147" candidate_uuid: "1d7717733275464198449a77396588d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1"
I20260927 10:53:44.253077 24258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a5adbccf904e668eb6441ba0a6d147" candidate_uuid: "1d7717733275464198449a77396588d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebb03ff175b4b1ab2a0df5097c84879"
I20260927 10:53:44.253155 23996 raft_consensus.cc:3037] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:44.253198 24258 raft_consensus.cc:3037] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:44.254190 23996 raft_consensus.cc:2454] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d7717733275464198449a77396588d6 in term 1.
I20260927 10:53:44.254207 24258 raft_consensus.cc:2454] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d7717733275464198449a77396588d6 in term 1.
I20260927 10:53:44.254537 24060 leader_election.cc:302] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 123f4fb45eff4c3c93f002a2c52ae4f1, 1d7717733275464198449a77396588d6, 8ebb03ff175b4b1ab2a0df5097c84879; no voters:
I20260927 10:53:44.254719 24329 raft_consensus.cc:2781] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:44.254925 24329 raft_consensus.cc:686] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 1 LEADER]: Becoming Leader. State: Replica: 1d7717733275464198449a77396588d6, State: Running, Role: LEADER
I20260927 10:53:44.255281 24329 consensus_queue.cc:237] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } }
I20260927 10:53:44.258615 23850 catalog_manager.cc:5495] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d7717733275464198449a77396588d6 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "1d7717733275464198449a77396588d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7717733275464198449a77396588d6" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 44313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:44.276149 24317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.272491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.6.31.130:44313)
0927 10:53:44.272620 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:44.272625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:44.272639 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:53:44.272812 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:44.272815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:44.272874 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:44.272990 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:44.273276 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.273795 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.274689 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:44.274908 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:44.275075 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:44.275333 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:44.275749 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:44.275888 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:44.275928 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 10:53:44.275978 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:53:44.282020 23996 raft_consensus.cc:1261] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1 [term 1 FOLLOWER]: Refusing update from remote peer 1d7717733275464198449a77396588d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:44.282058 24258 raft_consensus.cc:1261] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [term 1 FOLLOWER]: Refusing update from remote peer 1d7717733275464198449a77396588d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:44.282717 24331 consensus_queue.cc:1035] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebb03ff175b4b1ab2a0df5097c84879" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 38177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:44.282922 24329 consensus_queue.cc:1035] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123f4fb45eff4c3c93f002a2c52ae4f1" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:44.299396 23850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:44.299573 23850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:44.302204 23850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8743a7442f69402b8ebcb4aa70188f92: Sending DeleteTablet for 3 replicas of tablet 52a5adbccf904e668eb6441ba0a6d147
I20260927 10:53:44.303050 23976 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a5adbccf904e668eb6441ba0a6d147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:44 UTC) from {username='slave'} at 127.0.0.1:47414
I20260927 10:53:44.303289 24238 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a5adbccf904e668eb6441ba0a6d147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:44 UTC) from {username='slave'} at 127.0.0.1:60914
I20260927 10:53:44.303614 24107 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a5adbccf904e668eb6441ba0a6d147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:44 UTC) from {username='slave'} at 127.0.0.1:41636
I20260927 10:53:44.303685 24344 tablet_replica.cc:330] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1: stopping tablet replica
I20260927 10:53:44.303718 24345 tablet_replica.cc:330] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: stopping tablet replica
I20260927 10:53:44.304042 24344 raft_consensus.cc:2229] T 52a5adbccf904e668eb6441ba0a6d147 P 123f4fb45eff4c3c93f002a2c52ae4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:44.304042 24345 raft_consensus.cc:2229] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:44.304396 24345 raft_consensus.cc:2258] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:44.304647 24346 tablet_replica.cc:330] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6: stopping tablet replica
I20260927 10:53:44.304970 24346 raft_consensus.cc:2229] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:44.305425 24346 raft_consensus.cc:2258] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:44.305431 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 23914
I20260927 10:53:44.306830 24345 ts_tablet_manager.cc:1905] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:44.310606 24345 ts_tablet_manager.cc:1918] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:44.310721 24345 log.cc:1196] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/52a5adbccf904e668eb6441ba0a6d147
I20260927 10:53:44.311071 24345 ts_tablet_manager.cc:1939] T 52a5adbccf904e668eb6441ba0a6d147 P 8ebb03ff175b4b1ab2a0df5097c84879: Deleting consensus metadata
I20260927 10:53:44.312553 23838 catalog_manager.cc:4841] TS 8ebb03ff175b4b1ab2a0df5097c84879 (127.6.31.131:38177): tablet 52a5adbccf904e668eb6441ba0a6d147 (table test-table [id=970d75df45aa4e06aa2d4f138efaf57d]) successfully deleted
I20260927 10:53:44.313138 24346 ts_tablet_manager.cc:1905] T 52a5adbccf904e668eb6441ba0a6d147 P 1d7717733275464198449a77396588d6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:53:44.313318 23841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:35233 (error 108)
I20260927 10:53:44.313481 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 24045
I20260927 10:53:44.315809 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.315517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:35233 (local address 127.0.0.1:47420)
0927 10:53:44.315666 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.315730 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 10:53:44.315973 23841 catalog_manager.cc:4584] TS 123f4fb45eff4c3c93f002a2c52ae4f1 (127.6.31.129:35233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 52a5adbccf904e668eb6441ba0a6d147: Network error: Client connection negotiation failed: client connection to 127.6.31.129:35233: connect: Connection refused (error 111)
I20260927 10:53:44.319849 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 24176
I20260927 10:53:44.320678 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.319934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:44313 (local address 127.0.0.1:41658)
0927 10:53:44.320507 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.320575 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.130:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468}
I20260927 10:53:44.325392 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 23823
2026-09-27T10:53:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1098 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:44 dist-test-slave-c60x krb5kdc[24350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:44 dist-test-slave-c60x krb5kdc[24350](info): set up 2 sockets
Sep 27 10:53:44 dist-test-slave-c60x krb5kdc[24350](info): commencing operation
krb5kdc: starting...
W20260927 10:53:45.383013 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:53:45 dist-test-slave-c60x krb5kdc[24350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:53:45.511500 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38525
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:38525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:45.620127 24363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:45.620402 24363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:45.620445 24363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:45.624716 24363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:45.624788 24363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:45.624809 24363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:45.624825 24363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:45.624840 24363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:45.630235 24363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:38525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:45.631349 24363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:45.632373 24363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:45.641268 24368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:45.641399 24363 server_base.cc:1034] running on GCE node
W20260927 10:53:45.641261 24369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:45.641584 24371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:45.642145 24363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:45.643396 24363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:45.644582 24363 hybrid_clock.cc:648] HybridClock initialized: now 1790506425644542 us; error 53 us; skew 500 ppm
Sep 27 10:53:45 dist-test-slave-c60x krb5kdc[24350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506425, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:45.647516 24363 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:53:45.648474 24363 webserver.cc:460] Webserver started at http://127.6.31.190:35443/ using document root <none> and password file <none>
I20260927 10:53:45.649063 24363 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:45.649161 24363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:45.649396 24363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:45.651527 24363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9"
format_stamp: "Formatted at 2026-09-27 10:53:45 on dist-test-slave-c60x"
server_key: "1ea23a4de11de64e1b93dbf1dffbbbe3"
server_key_iv: "58dbeae77af793e73476bcc784441c2b"
server_key_version: "encryptionkey@0"
I20260927 10:53:45.652097 24363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9"
format_stamp: "Formatted at 2026-09-27 10:53:45 on dist-test-slave-c60x"
server_key: "1ea23a4de11de64e1b93dbf1dffbbbe3"
server_key_iv: "58dbeae77af793e73476bcc784441c2b"
server_key_version: "encryptionkey@0"
I20260927 10:53:45.655884 24363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:53:45.658404 24378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:45.659833 24363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:53:45.659986 24363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9"
format_stamp: "Formatted at 2026-09-27 10:53:45 on dist-test-slave-c60x"
server_key: "1ea23a4de11de64e1b93dbf1dffbbbe3"
server_key_iv: "58dbeae77af793e73476bcc784441c2b"
server_key_version: "encryptionkey@0"
I20260927 10:53:45.660113 24363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:45.673372 24363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:45.676723 24363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:45.676954 24363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:45.684310 24363 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:38525
I20260927 10:53:45.684330 24430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:38525 every 8 connection(s)
I20260927 10:53:45.685274 24363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:45.688045 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 24363
I20260927 10:53:45.688189 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:45.688249 24431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:45.688522 6270 external_mini_cluster.cc:1428] Setting key 34881067cb37cc6431b9f1dbf5d191c9
I20260927 10:53:45.694840 24431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9: Bootstrap starting.
Sep 27 10:53:45 dist-test-slave-c60x krb5kdc[24350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:45.697525 24431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:45.698269 24431 log.cc:824] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:45.700349 24431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9: No bootstrap required, opened a new log
I20260927 10:53:45.702836 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.690020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.6.31.190:38525)
0927 10:53:45.690479 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:45.690495 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:45.690528 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:53:45.691206 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:45.691223 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:45.691542 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:45.691871 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:45.692683 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.693674 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.694529 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.695068 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.695287 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.698039 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:45.698044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:45.698059 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:45.698088 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:45.699961 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:45.700447 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:45.700453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:45.700459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:45.700601 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:45.700902 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:45.700907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:45.700909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:45.701313 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:45.701506 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:45.701811 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 10:53:45.702108 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":150,"threads_started":1}
I20260927 10:53:45.704324 24431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38525 } }
I20260927 10:53:45.704607 24431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:45.704661 24431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8e18c8643f4740a3e2ffe6e7aa17b9, State: Initialized, Role: FOLLOWER
I20260927 10:53:45.705135 24431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38525 } }
I20260927 10:53:45.705303 24431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:45.705358 24431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:45.705435 24431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:45.706372 24431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38525 } }
I20260927 10:53:45.706748 24431 leader_election.cc:302] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b8e18c8643f4740a3e2ffe6e7aa17b9; no voters:
I20260927 10:53:45.707140 24431 leader_election.cc:288] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:45.707270 24436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:45.707504 24436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1b8e18c8643f4740a3e2ffe6e7aa17b9, State: Running, Role: LEADER
I20260927 10:53:45.707854 24436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38525 } }
I20260927 10:53:45.708446 24431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:45.709614 24437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38525 } } }
I20260927 10:53:45.709653 24438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b8e18c8643f4740a3e2ffe6e7aa17b9. Latest consensus state: current_term: 1 leader_uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e18c8643f4740a3e2ffe6e7aa17b9" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38525 } } }
I20260927 10:53:45.709801 24438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:45.710228 24445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:45.710119 24437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:45.713327 24445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:45.719362 24445 catalog_manager.cc:1347] Generated new cluster ID: e0d5f194a9104bff82ece93394b1a115
I20260927 10:53:45.719447 24445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:45.740326 24445 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:45.741345 24445 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:45.746332 24445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9: Generated new TSK 0
I20260927 10:53:45.747056 24445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:53:45.779634 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38525
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:45.893200 24457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:45.893524 24457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:45.893618 24457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:45.897802 24457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:45.897902 24457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:45.897996 24457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:53:45.903438 24457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:45.904572 24457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:45.905622 24457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:45.915448 24462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:45.915441 24463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:45.915491 24465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:45.915612 24457 server_base.cc:1034] running on GCE node
I20260927 10:53:45.916131 24457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:45.916734 24457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:45.917949 24457 hybrid_clock.cc:648] HybridClock initialized: now 1790506425917918 us; error 41 us; skew 500 ppm
Sep 27 10:53:45 dist-test-slave-c60x krb5kdc[24350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506425, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:45.921075 24457 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:53:45.922035 24457 webserver.cc:460] Webserver started at http://127.6.31.129:42207/ using document root <none> and password file <none>
I20260927 10:53:45.922600 24457 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:45.922665 24457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:45.922876 24457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:45.924988 24457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "d6faecdafe284d3cadfabc20aaf7e936"
format_stamp: "Formatted at 2026-09-27 10:53:45 on dist-test-slave-c60x"
server_key: "57526c58fd708a29220c98fca8dc4d09"
server_key_iv: "0704681d54e22efce67c450d198cd91b"
server_key_version: "encryptionkey@0"
I20260927 10:53:45.925560 24457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "d6faecdafe284d3cadfabc20aaf7e936"
format_stamp: "Formatted at 2026-09-27 10:53:45 on dist-test-slave-c60x"
server_key: "57526c58fd708a29220c98fca8dc4d09"
server_key_iv: "0704681d54e22efce67c450d198cd91b"
server_key_version: "encryptionkey@0"
I20260927 10:53:45.929440 24457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:53:45.931995 24472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:45.933250 24457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:45.933372 24457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "d6faecdafe284d3cadfabc20aaf7e936"
format_stamp: "Formatted at 2026-09-27 10:53:45 on dist-test-slave-c60x"
server_key: "57526c58fd708a29220c98fca8dc4d09"
server_key_iv: "0704681d54e22efce67c450d198cd91b"
server_key_version: "encryptionkey@0"
I20260927 10:53:45.933485 24457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:45.950337 24457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:45.953640 24457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:45.953864 24457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:45.954499 24457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:45.955397 24457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:45.955446 24457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:45.955515 24457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:45.955546 24457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:45.965015 24457 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:36055
I20260927 10:53:45.965109 24585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:36055 every 8 connection(s)
I20260927 10:53:45.965992 24457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:45.966286 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 24457
I20260927 10:53:45.966427 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:45.966755 6270 external_mini_cluster.cc:1428] Setting key 7d784672d75aa0030826b2d682f66723
Sep 27 10:53:45 dist-test-slave-c60x krb5kdc[24350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506425, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:53:45.979907 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.967904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:43091 (local address 127.6.31.190:38525)
0927 10:53:45.968053 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:45.968057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:45.969149 (+ 1092us) server_negotiation.cc:407] Connection header received
0927 10:53:45.970017 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:45.970020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:45.970086 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:45.970186 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:45.971422 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.971965 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.973130 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.973367 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.973479 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.976169 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:45.976173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:45.976175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:45.976199 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:45.977736 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:45.978264 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:45.978268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:45.978270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:45.978350 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:45.978707 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:45.978712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:45.978714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:45.978910 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:45.979019 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:45.979605 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 10:53:45.979736 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:53:45.980734 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.968387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38525 (local address 127.6.31.129:43091)
0927 10:53:45.968916 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.968969 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:45.969809 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:45.970360 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:45.970367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:45.970742 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:45.971278 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:45.971293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.972097 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.972101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:45.972995 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:45.973004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.973521 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.973524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:45.973699 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.974320 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:45.974337 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:45.975994 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:45.977866 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:45.977872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:45.977887 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:45.978155 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:45.978447 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:45.978451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:45.978453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:45.978608 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:45.979019 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:45.979027 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:45.979341 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 10:53:45.979545 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 10:53:45.979895 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":143,"threads_started":1}
I20260927 10:53:45.982048 24586 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38525
I20260927 10:53:45.982307 24586 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:45.982811 24586 heartbeater.cc:511] Master 127.6.31.190:38525 requested a full tablet report, sending...
I20260927 10:53:45.984371 24394 ts_manager.cc:194] Registered new tserver with Master: d6faecdafe284d3cadfabc20aaf7e936 (127.6.31.129:36055)
I20260927 10:53:45.985486 24394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:43091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:53:45.997948 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38525
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:46.113335 24591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:46.113583 24591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:46.113660 24591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:46.117908 24591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:46.117977 24591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:46.118077 24591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:53:46.123606 24591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:46.124670 24591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:46.125687 24591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:46.135332 24597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:46.135332 24596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:46.135332 24599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:46.135772 24591 server_base.cc:1034] running on GCE node
I20260927 10:53:46.136142 24591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:46.136776 24591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:46.137938 24591 hybrid_clock.cc:648] HybridClock initialized: now 1790506426137919 us; error 38 us; skew 500 ppm
Sep 27 10:53:46 dist-test-slave-c60x krb5kdc[24350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:46.140832 24591 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:53:46.141724 24591 webserver.cc:460] Webserver started at http://127.6.31.130:42829/ using document root <none> and password file <none>
I20260927 10:53:46.142510 24591 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:46.142586 24591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:46.142764 24591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:46.144940 24591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "246a3232731446d68a2d6c45e45d1282"
format_stamp: "Formatted at 2026-09-27 10:53:46 on dist-test-slave-c60x"
server_key: "25a5d746d99d7d49fecdafcf16017e99"
server_key_iv: "33d824c1485346982f629a8b11d9aaa2"
server_key_version: "encryptionkey@0"
I20260927 10:53:46.145618 24591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "246a3232731446d68a2d6c45e45d1282"
format_stamp: "Formatted at 2026-09-27 10:53:46 on dist-test-slave-c60x"
server_key: "25a5d746d99d7d49fecdafcf16017e99"
server_key_iv: "33d824c1485346982f629a8b11d9aaa2"
server_key_version: "encryptionkey@0"
I20260927 10:53:46.149381 24591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:46.151701 24606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:46.152863 24591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:46.152997 24591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "246a3232731446d68a2d6c45e45d1282"
format_stamp: "Formatted at 2026-09-27 10:53:46 on dist-test-slave-c60x"
server_key: "25a5d746d99d7d49fecdafcf16017e99"
server_key_iv: "33d824c1485346982f629a8b11d9aaa2"
server_key_version: "encryptionkey@0"
I20260927 10:53:46.153127 24591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:46.180972 24591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:46.183861 24591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:46.184110 24591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:46.184706 24591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:46.185527 24591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:46.185599 24591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:46.185664 24591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:46.185711 24591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:46.194873 24591 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:42303
I20260927 10:53:46.194903 24719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:42303 every 8 connection(s)
I20260927 10:53:46.195792 24591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:46.204609 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 24591
I20260927 10:53:46.204730 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
Sep 27 10:53:46 dist-test-slave-c60x krb5kdc[24350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:46.205008 6270 external_mini_cluster.cc:1428] Setting key 0f8ffd6cf3b75763d4e785e53c2b54b3
I20260927 10:53:46.209064 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.197688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:52879 (local address 127.6.31.190:38525)
0927 10:53:46.197829 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.197833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.198657 (+ 824us) server_negotiation.cc:407] Connection header received
0927 10:53:46.199555 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.199558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.199607 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.199680 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:46.200861 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.201378 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.202487 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.202701 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.202778 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.205411 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:46.205415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:46.205417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:46.205445 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:46.206986 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:46.207519 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:46.207524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:46.207526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:46.207616 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:46.207963 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:46.207966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:46.207967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:46.208155 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:46.208266 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.208782 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.208908 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:53:46.210261 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.198060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38525 (local address 127.6.31.130:52879)
0927 10:53:46.198503 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:46.198543 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:46.199371 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:46.199890 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:46.199906 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:46.200254 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:46.200735 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.200747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.201501 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.201504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.202373 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.202382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.202839 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.202842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.202974 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.203611 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:46.203627 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:46.205250 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:46.207136 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:46.207142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:46.207156 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:46.207414 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:46.207714 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:46.207718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:46.207720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:46.207859 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:46.208264 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:46.208271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:46.208515 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 10:53:46.208703 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 10:53:46.209382 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260927 10:53:46.211532 24720 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38525
I20260927 10:53:46.211779 24720 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:46.212260 24720 heartbeater.cc:511] Master 127.6.31.190:38525 requested a full tablet report, sending...
I20260927 10:53:46.213299 24394 ts_manager.cc:194] Registered new tserver with Master: 246a3232731446d68a2d6c45e45d1282 (127.6.31.130:42303)
I20260927 10:53:46.213867 24394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:52879
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:53:46.234658 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38525
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:46.347229 24725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:46.347506 24725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:46.347580 24725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:46.352038 24725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:46.352119 24725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:46.352202 24725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:53:46.357579 24725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:46.358644 24725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:46.359798 24725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:46.369376 24733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:46.369522 24725 server_base.cc:1034] running on GCE node
W20260927 10:53:46.369376 24731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:46.369431 24730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:46.370065 24725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:46.370700 24725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:46.371888 24725 hybrid_clock.cc:648] HybridClock initialized: now 1790506426371846 us; error 59 us; skew 500 ppm
Sep 27 10:53:46 dist-test-slave-c60x krb5kdc[24350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:46.374665 24725 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:53:46.375650 24725 webserver.cc:460] Webserver started at http://127.6.31.131:44455/ using document root <none> and password file <none>
I20260927 10:53:46.376226 24725 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:46.376318 24725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:46.376546 24725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:46.378569 24725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "79fc52a05ee245a2a622ea85b484e033"
format_stamp: "Formatted at 2026-09-27 10:53:46 on dist-test-slave-c60x"
server_key: "eb0655b27e8963e650155c73ef91133e"
server_key_iv: "df408395b8eb6fa14220a2b40ef6578a"
server_key_version: "encryptionkey@0"
I20260927 10:53:46.379204 24725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "79fc52a05ee245a2a622ea85b484e033"
format_stamp: "Formatted at 2026-09-27 10:53:46 on dist-test-slave-c60x"
server_key: "eb0655b27e8963e650155c73ef91133e"
server_key_iv: "df408395b8eb6fa14220a2b40ef6578a"
server_key_version: "encryptionkey@0"
I20260927 10:53:46.382789 24725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:46.385396 24740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:46.386636 24725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:46.386785 24725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "79fc52a05ee245a2a622ea85b484e033"
format_stamp: "Formatted at 2026-09-27 10:53:46 on dist-test-slave-c60x"
server_key: "eb0655b27e8963e650155c73ef91133e"
server_key_iv: "df408395b8eb6fa14220a2b40ef6578a"
server_key_version: "encryptionkey@0"
I20260927 10:53:46.386902 24725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:46.404503 24725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:46.407441 24725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:46.407648 24725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:46.408334 24725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:46.409245 24725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:46.409309 24725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:46.409411 24725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:46.409447 24725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:46.418476 24725 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:43471
I20260927 10:53:46.418494 24853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:43471 every 8 connection(s)
I20260927 10:53:46.419487 24725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:46.421373 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 24725
I20260927 10:53:46.421509 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:46.421795 6270 external_mini_cluster.cc:1428] Setting key c12c7f9854a349cc7a3f7659c5bb3914
Sep 27 10:53:46 dist-test-slave-c60x krb5kdc[24350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:46.432842 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.421408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:42057 (local address 127.6.31.190:38525)
0927 10:53:46.421591 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.421596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.422810 (+ 1214us) server_negotiation.cc:407] Connection header received
0927 10:53:46.423921 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.423925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.423975 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.424084 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:46.425173 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.425688 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.426772 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.426977 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.427152 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.429521 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:46.429523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:46.429526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:46.429547 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:46.430916 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:46.431492 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:46.431497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:46.431498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:46.431597 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:46.431920 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:46.431923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:46.431924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:46.432099 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:46.432169 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.432623 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.432723 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:53:46.433616 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.421819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38525 (local address 127.6.31.131:42057)
0927 10:53:46.422627 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:46.422682 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:46.423711 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:46.424236 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:46.424242 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:46.424580 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:46.425044 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.425056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.425791 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.425793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.426669 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.426677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.427175 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.427182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.427365 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.427929 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:46.427946 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:46.429349 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:46.431030 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:46.431034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:46.431045 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:46.431376 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:46.431703 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:46.431705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:46.431706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:46.431836 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:46.432183 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:46.432188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:46.432401 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 10:53:46.432543 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 10:53:46.432784 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":122,"threads_started":1}
I20260927 10:53:46.434717 24854 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38525
I20260927 10:53:46.434957 24854 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:46.435473 24854 heartbeater.cc:511] Master 127.6.31.190:38525 requested a full tablet report, sending...
I20260927 10:53:46.436450 24394 ts_manager.cc:194] Registered new tserver with Master: 79fc52a05ee245a2a622ea85b484e033 (127.6.31.131:43471)
I20260927 10:53:46.437062 24394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:42057
I20260927 10:53:46.447595 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:46.457396 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.449609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.6.31.190:38525)
0927 10:53:46.449740 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.449744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.449939 (+ 195us) server_negotiation.cc:407] Connection header received
0927 10:53:46.450017 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.450019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.450068 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.450132 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:46.450766 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.451393 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.452250 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.452459 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.452543 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.454054 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:46.454057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:46.454090 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:46.454114 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:46.455472 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:46.456022 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:46.456025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:46.456026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:46.456129 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:46.456483 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:46.456485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:46.456486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:46.456689 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:46.456806 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.457102 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.457220 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:53:46.460420 24394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:46.463433 24394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:46.478044 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.472258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:42303 (local address 127.0.0.1:59122)
0927 10:53:46.472623 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:46.472676 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:46.472807 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:46.473649 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:46.473655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:46.473679 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:46.473813 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.473825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.475105 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.475110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.476430 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.476440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.477511 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.477514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.477615 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.477650 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:53:46.477780 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 10:53:46.477896 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":4,"thread_start_us":102,"threads_started":1}
I20260927 10:53:46.478055 24866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.472444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.6.31.130:42303)
0927 10:53:46.472916 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.472929 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.472953 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:53:46.473035 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.473042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.473260 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.473416 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:46.473958 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.474954 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.476564 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.477386 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.477506 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.477648 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.477809 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.477896 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":132,"threads_started":1}
I20260927 10:53:46.478266 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.473301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:43471 (local address 127.0.0.1:48136)
0927 10:53:46.473703 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:46.473717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:46.473789 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:46.474434 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:46.474438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:46.474453 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:46.474522 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.474527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.475871 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.475875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.476856 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.476863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.477850 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.477854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.477948 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.477987 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 10:53:46.478039 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 10:53:46.478107 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260927 10:53:46.478824 24870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.473714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.6.31.131:43471)
0927 10:53:46.474044 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.474057 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.474076 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:53:46.474151 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.474156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.474319 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.474464 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:46.474600 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.475707 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.476982 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.477707 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.478443 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.478571 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.478619 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.478699 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":142,"threads_started":1}
I20260927 10:53:46.478813 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.472991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36055 (local address 127.0.0.1:54840)
0927 10:53:46.473557 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:46.473572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:46.473671 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:46.474187 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:46.474190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:46.474208 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:46.474285 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.474290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.475724 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.475729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.477118 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.477129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.478408 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.478412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.478506 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.478526 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:53:46.478606 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:53:46.478669 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":41,"thread_start_us":92,"threads_started":1}
I20260927 10:53:46.478880 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.473169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.6.31.129:36055)
0927 10:53:46.473571 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.473585 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.473769 (+ 184us) server_negotiation.cc:407] Connection header received
0927 10:53:46.473847 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.473855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.474036 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.474185 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:46.474381 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.475576 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.477259 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.478121 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.478215 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.478346 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.478654 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.478744 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":94,"threads_started":1}
I20260927 10:53:46.481336 24654 tablet_service.cc:1467] Processing CreateTablet for tablet b647fb757aa84a9ea53c9aee7ac44a0f (DEFAULT_TABLE table=test-table [id=15fe27a655054859934e677397288332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:46.481550 24520 tablet_service.cc:1467] Processing CreateTablet for tablet b647fb757aa84a9ea53c9aee7ac44a0f (DEFAULT_TABLE table=test-table [id=15fe27a655054859934e677397288332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:46.482338 24654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b647fb757aa84a9ea53c9aee7ac44a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:46.482446 24520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b647fb757aa84a9ea53c9aee7ac44a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:46.488366 24872 tablet_bootstrap.cc:492] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282: Bootstrap starting.
I20260927 10:53:46.488194 24871 tablet_bootstrap.cc:492] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936: Bootstrap starting.
I20260927 10:53:46.490912 24871 tablet_bootstrap.cc:654] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:46.491075 24872 tablet_bootstrap.cc:654] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:46.490844 24788 tablet_service.cc:1467] Processing CreateTablet for tablet b647fb757aa84a9ea53c9aee7ac44a0f (DEFAULT_TABLE table=test-table [id=15fe27a655054859934e677397288332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:46.491780 24871 log.cc:824] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:46.491891 24872 log.cc:824] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:46.491966 24788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b647fb757aa84a9ea53c9aee7ac44a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:46.493778 24871 tablet_bootstrap.cc:492] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936: No bootstrap required, opened a new log
I20260927 10:53:46.493968 24871 ts_tablet_manager.cc:1397] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260927 10:53:46.495609 24872 tablet_bootstrap.cc:492] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282: No bootstrap required, opened a new log
I20260927 10:53:46.495823 24872 ts_tablet_manager.cc:1397] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 10:53:46.498476 24872 raft_consensus.cc:348] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } }
I20260927 10:53:46.498498 24871 raft_consensus.cc:348] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } }
I20260927 10:53:46.498725 24871 raft_consensus.cc:374] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:46.498725 24872 raft_consensus.cc:374] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:46.498783 24872 raft_consensus.cc:729] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246a3232731446d68a2d6c45e45d1282, State: Initialized, Role: FOLLOWER
I20260927 10:53:46.498785 24871 raft_consensus.cc:729] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6faecdafe284d3cadfabc20aaf7e936, State: Initialized, Role: FOLLOWER
I20260927 10:53:46.499226 24875 tablet_bootstrap.cc:492] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: Bootstrap starting.
I20260927 10:53:46.499279 24872 consensus_queue.cc:260] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } }
I20260927 10:53:46.499279 24871 consensus_queue.cc:260] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } }
I20260927 10:53:46.500274 24871 ts_tablet_manager.cc:1428] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:53:46.500273 24872 ts_tablet_manager.cc:1428] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 10:53:46.500636 24720 heartbeater.cc:503] Master 127.6.31.190:38525 was elected leader, sending a full tablet report...
I20260927 10:53:46.500941 24586 heartbeater.cc:503] Master 127.6.31.190:38525 was elected leader, sending a full tablet report...
I20260927 10:53:46.501874 24875 tablet_bootstrap.cc:654] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:46.503175 24875 log.cc:824] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:46.505304 24875 tablet_bootstrap.cc:492] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: No bootstrap required, opened a new log
I20260927 10:53:46.505532 24875 ts_tablet_manager.cc:1397] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:53:46.508448 24875 raft_consensus.cc:348] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } }
I20260927 10:53:46.508783 24875 raft_consensus.cc:374] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:46.508842 24875 raft_consensus.cc:729] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79fc52a05ee245a2a622ea85b484e033, State: Initialized, Role: FOLLOWER
I20260927 10:53:46.509231 24875 consensus_queue.cc:260] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } }
I20260927 10:53:46.509956 24854 heartbeater.cc:503] Master 127.6.31.190:38525 was elected leader, sending a full tablet report...
I20260927 10:53:46.510335 24875 ts_tablet_manager.cc:1428] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:53:46.546602 24876 raft_consensus.cc:482] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:46.546850 24876 raft_consensus.cc:504] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } }
I20260927 10:53:46.548105 24876 leader_election.cc:288] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6faecdafe284d3cadfabc20aaf7e936 (127.6.31.129:36055), 79fc52a05ee245a2a622ea85b484e033 (127.6.31.131:43471)
I20260927 10:53:46.552825 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.548394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36055 (local address 127.6.31.130:46617)
0927 10:53:46.548522 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:46.548539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:46.548650 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:46.549095 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:46.549097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:46.549116 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:46.549267 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.549273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.550275 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.550278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.551527 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.551536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.552342 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.552346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.552454 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.552481 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:53:46.552574 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:53:46.552665 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 10:53:46.552964 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.548502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:46617 (local address 127.6.31.129:36055)
0927 10:53:46.548625 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.548629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.548644 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:46.548793 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.548796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.548850 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.548934 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:46.549426 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.550144 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.551670 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.552203 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.552298 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.552342 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.552653 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.552834 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:53:46.553359 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.548693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:43471 (local address 127.6.31.130:47109)
0927 10:53:46.549157 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:46.549177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:46.549275 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:46.549649 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:46.549651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:46.549667 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:46.549778 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.549783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.550886 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.550888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.551842 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:46.551852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.552781 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.552791 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:46.552906 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.552929 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:53:46.553138 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 10:53:46.553215 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260927 10:53:46.553367 24870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.548870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:47109 (local address 127.6.31.131:43471)
0927 10:53:46.549112 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.549118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.549258 (+ 140us) server_negotiation.cc:407] Connection header received
0927 10:53:46.549405 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.549409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.549471 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.549521 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:46.549919 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.550756 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.551988 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.552560 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.552673 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.552713 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.553165 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.553250 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 10:53:46.553658 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b647fb757aa84a9ea53c9aee7ac44a0f" candidate_uuid: "246a3232731446d68a2d6c45e45d1282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6faecdafe284d3cadfabc20aaf7e936" is_pre_election: true
I20260927 10:53:46.554023 24540 raft_consensus.cc:2454] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 246a3232731446d68a2d6c45e45d1282 in term 0.
I20260927 10:53:46.554064 24808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b647fb757aa84a9ea53c9aee7ac44a0f" candidate_uuid: "246a3232731446d68a2d6c45e45d1282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79fc52a05ee245a2a622ea85b484e033" is_pre_election: true
I20260927 10:53:46.554346 24808 raft_consensus.cc:2454] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 246a3232731446d68a2d6c45e45d1282 in term 0.
I20260927 10:53:46.554585 24608 leader_election.cc:302] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 246a3232731446d68a2d6c45e45d1282, d6faecdafe284d3cadfabc20aaf7e936; no voters:
I20260927 10:53:46.554900 24876 raft_consensus.cc:2781] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:46.554977 24876 raft_consensus.cc:482] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:46.555038 24876 raft_consensus.cc:3037] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:46.556133 24876 raft_consensus.cc:504] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } }
I20260927 10:53:46.556509 24876 leader_election.cc:288] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [CANDIDATE]: Term 1 election: Requested vote from peers d6faecdafe284d3cadfabc20aaf7e936 (127.6.31.129:36055), 79fc52a05ee245a2a622ea85b484e033 (127.6.31.131:43471)
I20260927 10:53:46.556949 24808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b647fb757aa84a9ea53c9aee7ac44a0f" candidate_uuid: "246a3232731446d68a2d6c45e45d1282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79fc52a05ee245a2a622ea85b484e033"
I20260927 10:53:46.557029 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b647fb757aa84a9ea53c9aee7ac44a0f" candidate_uuid: "246a3232731446d68a2d6c45e45d1282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6faecdafe284d3cadfabc20aaf7e936"
I20260927 10:53:46.557091 24808 raft_consensus.cc:3037] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:46.557168 24540 raft_consensus.cc:3037] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:46.558173 24808 raft_consensus.cc:2454] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 246a3232731446d68a2d6c45e45d1282 in term 1.
I20260927 10:53:46.558483 24540 raft_consensus.cc:2454] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 246a3232731446d68a2d6c45e45d1282 in term 1.
I20260927 10:53:46.558528 24608 leader_election.cc:302] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 246a3232731446d68a2d6c45e45d1282, 79fc52a05ee245a2a622ea85b484e033; no voters:
I20260927 10:53:46.558688 24876 raft_consensus.cc:2781] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:46.558928 24876 raft_consensus.cc:686] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 1 LEADER]: Becoming Leader. State: Replica: 246a3232731446d68a2d6c45e45d1282, State: Running, Role: LEADER
I20260927 10:53:46.559307 24876 consensus_queue.cc:237] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } }
I20260927 10:53:46.562541 24394 catalog_manager.cc:5495] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 reported cstate change: term changed from 0 to 1, leader changed from <none> to 246a3232731446d68a2d6c45e45d1282 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "246a3232731446d68a2d6c45e45d1282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a3232731446d68a2d6c45e45d1282" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:46.578409 24866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.574493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.6.31.130:42303)
0927 10:53:46.574811 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.574817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.574834 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:53:46.574924 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.574927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.575033 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.575207 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:46.575475 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.576007 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.576878 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.577107 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.577219 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.577537 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:46.577948 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:46.578111 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.578196 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.578256 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:53:46.584497 24540 raft_consensus.cc:1261] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936 [term 1 FOLLOWER]: Refusing update from remote peer 246a3232731446d68a2d6c45e45d1282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:46.584502 24808 raft_consensus.cc:1261] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [term 1 FOLLOWER]: Refusing update from remote peer 246a3232731446d68a2d6c45e45d1282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:46.585350 24876 consensus_queue.cc:1035] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79fc52a05ee245a2a622ea85b484e033" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:46.585614 24881 consensus_queue.cc:1035] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6faecdafe284d3cadfabc20aaf7e936" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:46.594846 24885 mvcc.cc:205] Tried to move back new op lower bound from 7333914323281899520 to 7333914323189497856. Current Snapshot: MvccSnapshot[applied={T|T < 7333914323281899520}]
I20260927 10:53:46.602943 24394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:46.603179 24394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:46.605644 24394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b8e18c8643f4740a3e2ffe6e7aa17b9: Sending DeleteTablet for 3 replicas of tablet b647fb757aa84a9ea53c9aee7ac44a0f
I20260927 10:53:46.606434 24654 tablet_service.cc:1514] Processing DeleteTablet for tablet b647fb757aa84a9ea53c9aee7ac44a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:46 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:59122
I20260927 10:53:46.606535 24520 tablet_service.cc:1514] Processing DeleteTablet for tablet b647fb757aa84a9ea53c9aee7ac44a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:46 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:54840
I20260927 10:53:46.606601 24788 tablet_service.cc:1514] Processing DeleteTablet for tablet b647fb757aa84a9ea53c9aee7ac44a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:46 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:48136
I20260927 10:53:46.607026 24896 tablet_replica.cc:330] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936: stopping tablet replica
I20260927 10:53:46.607196 24894 tablet_replica.cc:330] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282: stopping tablet replica
I20260927 10:53:46.607391 24896 raft_consensus.cc:2229] T b647fb757aa84a9ea53c9aee7ac44a0f P d6faecdafe284d3cadfabc20aaf7e936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:46.607549 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 24457
I20260927 10:53:46.607647 24894 raft_consensus.cc:2229] T b647fb757aa84a9ea53c9aee7ac44a0f P 246a3232731446d68a2d6c45e45d1282 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:46.609681 24895 tablet_replica.cc:330] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: stopping tablet replica
I20260927 10:53:46.609993 24895 raft_consensus.cc:2229] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:46.610286 24895 raft_consensus.cc:2258] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:46.611686 24895 ts_tablet_manager.cc:1905] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:46.613960 24895 ts_tablet_manager.cc:1918] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:46.614073 24895 log.cc:1196] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/b647fb757aa84a9ea53c9aee7ac44a0f
I20260927 10:53:46.614368 24895 ts_tablet_manager.cc:1939] T b647fb757aa84a9ea53c9aee7ac44a0f P 79fc52a05ee245a2a622ea85b484e033: Deleting consensus metadata
I20260927 10:53:46.615623 24380 catalog_manager.cc:4841] TS 79fc52a05ee245a2a622ea85b484e033 (127.6.31.131:43471): tablet b647fb757aa84a9ea53c9aee7ac44a0f (table test-table [id=15fe27a655054859934e677397288332]) successfully deleted
I20260927 10:53:46.615954 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 24591
W20260927 10:53:46.615926 24380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:36055 (error 108)
I20260927 10:53:46.618247 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.618000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36055 (local address 127.0.0.1:54856)
0927 10:53:46.618121 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:46.618176 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:53:46.618392 24380 catalog_manager.cc:4584] TS d6faecdafe284d3cadfabc20aaf7e936 (127.6.31.129:36055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b647fb757aa84a9ea53c9aee7ac44a0f: Network error: Client connection negotiation failed: client connection to 127.6.31.129:36055: connect: Connection refused (error 111)
I20260927 10:53:46.621824 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 24725
I20260927 10:53:46.627729 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.621973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:42303 (local address 127.0.0.1:59138)
0927 10:53:46.627529 (+ 5556us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:46.627620 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.130:42303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5443}
I20260927 10:53:46.627717 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 24363
2026-09-27T10:53:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T10:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:46Z Disabled control of system clock
I20260927 10:53:46.657842 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:41815
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:41815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:53:46.766934 24901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:46.767237 24901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:46.767274 24901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:46.771152 24901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:46.771200 24901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:46.771216 24901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:46.771231 24901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:46.771246 24901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:46.776072 24901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:41815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:41815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:46.777179 24901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:46.778486 24901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:46.786996 24907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:46.787019 24906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:46.787149 24901 server_base.cc:1034] running on GCE node
W20260927 10:53:46.787070 24909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:46.787880 24901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:46.789001 24901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:46.790226 24901 hybrid_clock.cc:648] HybridClock initialized: now 1790506426790212 us; error 38 us; skew 500 ppm
I20260927 10:53:46.792349 24901 webserver.cc:460] Webserver started at http://127.6.31.190:34587/ using document root <none> and password file <none>
I20260927 10:53:46.792963 24901 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:46.793062 24901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:46.793303 24901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:46.795454 24901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56"
format_stamp: "Formatted at 2026-09-27 10:53:46 on dist-test-slave-c60x"
server_key: "1a848fbee35465f742d84d59534bdea2"
server_key_iv: "bbde04c4be9df29daad7dded0b50d612"
server_key_version: "encryptionkey@0"
I20260927 10:53:46.796031 24901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56"
format_stamp: "Formatted at 2026-09-27 10:53:46 on dist-test-slave-c60x"
server_key: "1a848fbee35465f742d84d59534bdea2"
server_key_iv: "bbde04c4be9df29daad7dded0b50d612"
server_key_version: "encryptionkey@0"
I20260927 10:53:46.799772 24901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:46.802204 24915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:46.803525 24901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:46.803675 24901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56"
format_stamp: "Formatted at 2026-09-27 10:53:46 on dist-test-slave-c60x"
server_key: "1a848fbee35465f742d84d59534bdea2"
server_key_iv: "bbde04c4be9df29daad7dded0b50d612"
server_key_version: "encryptionkey@0"
I20260927 10:53:46.803790 24901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:46.840153 24901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:46.840955 24901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:46.841180 24901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:46.848531 24901 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:41815
I20260927 10:53:46.848531 24967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:41815 every 8 connection(s)
I20260927 10:53:46.849555 24901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:46.852732 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:46.854149 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 24901
I20260927 10:53:46.854281 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:46.854583 6270 external_mini_cluster.cc:1428] Setting key 30aea594c97e4fdd68f267737961f488
I20260927 10:53:46.858839 24968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56: Bootstrap starting.
I20260927 10:53:46.861194 24968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:46.862052 24968 log.cc:824] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:46.864212 24968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56: No bootstrap required, opened a new log
I20260927 10:53:46.864575 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:46.856088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.6.31.190:41815)
0927 10:53:46.856730 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:46.856745 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:46.856779 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:53:46.857421 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:46.857437 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:46.857807 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:46.858141 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:46.858551 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.859606 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.860498 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:46.861250 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:46.861448 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:46.862601 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:46.862607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:46.862625 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:46.863198 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:46.863255 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:46.863276 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:46.863354 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:46.863512 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 10:53:46.863790 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":135,"threads_started":1}
I20260927 10:53:46.868134 24968 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 41815 } }
I20260927 10:53:46.868381 24968 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:46.868474 24968 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae11ffa1bee54bb5b4e11ccff4a8ff56, State: Initialized, Role: FOLLOWER
I20260927 10:53:46.868919 24968 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 41815 } }
I20260927 10:53:46.869067 24968 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:46.869151 24968 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:46.869247 24968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:46.870139 24968 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 41815 } }
I20260927 10:53:46.870472 24968 leader_election.cc:302] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae11ffa1bee54bb5b4e11ccff4a8ff56; no voters:
I20260927 10:53:46.870769 24968 leader_election.cc:288] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:46.870944 24973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:46.871197 24973 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [term 1 LEADER]: Becoming Leader. State: Replica: ae11ffa1bee54bb5b4e11ccff4a8ff56, State: Running, Role: LEADER
I20260927 10:53:46.871472 24973 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 41815 } }
I20260927 10:53:46.872048 24968 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:46.873169 24975 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae11ffa1bee54bb5b4e11ccff4a8ff56. Latest consensus state: current_term: 1 leader_uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 41815 } } }
I20260927 10:53:46.873303 24975 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:46.873641 24974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae11ffa1bee54bb5b4e11ccff4a8ff56" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 41815 } } }
I20260927 10:53:46.873732 24974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:46.873759 24982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:46.876995 24982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:46.883157 24982 catalog_manager.cc:1347] Generated new cluster ID: 9fb0baa1d61d465fb6a86d2d0267e17c
I20260927 10:53:46.883231 24982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:46.888895 24982 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:46.890071 24982 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:46.898403 24982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56: Generated new TSK 0
I20260927 10:53:46.899219 24982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:46.903695 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:41815
--builtin_ntp_servers=127.6.31.148:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:53:47.017971 24992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:47.018229 24992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:47.018267 24992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:47.022414 24992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:47.022476 24992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:47.022532 24992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:53:47.027259 24992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:41815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:47.028265 24992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:47.029644 24992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:47.039325 24992 server_base.cc:1034] running on GCE node
W20260927 10:53:47.039276 24998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.039318 24997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.039278 25000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.039953 24992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:47.040548 24992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:47.041728 24992 hybrid_clock.cc:648] HybridClock initialized: now 1790506427041711 us; error 32 us; skew 500 ppm
I20260927 10:53:47.043746 24992 webserver.cc:460] Webserver started at http://127.6.31.129:35757/ using document root <none> and password file <none>
I20260927 10:53:47.044318 24992 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:47.044379 24992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:47.044643 24992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:47.046690 24992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "c824caf5762545fdbc6c75568a24e03c"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-c60x"
server_key: "c12d0c77f4745e2c6e1bcf5e25b867c8"
server_key_iv: "d892db6806a96b934c31c6eac92ebbda"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.047302 24992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "c824caf5762545fdbc6c75568a24e03c"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-c60x"
server_key: "c12d0c77f4745e2c6e1bcf5e25b867c8"
server_key_iv: "d892db6806a96b934c31c6eac92ebbda"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.050976 24992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:47.053361 25006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.054558 24992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:47.054693 24992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "c824caf5762545fdbc6c75568a24e03c"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-c60x"
server_key: "c12d0c77f4745e2c6e1bcf5e25b867c8"
server_key_iv: "d892db6806a96b934c31c6eac92ebbda"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.054809 24992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:47.077735 24992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:47.078503 24992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:47.078719 24992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:47.079402 24992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:47.080437 24992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:47.080518 24992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.080588 24992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:47.080634 24992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.090127 24992 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:41583
I20260927 10:53:47.090157 25119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:41583 every 8 connection(s)
I20260927 10:53:47.091063 24992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:47.100364 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 24992
I20260927 10:53:47.100536 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:47.100647 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.092941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:40727 (local address 127.6.31.190:41815)
0927 10:53:47.093081 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.093085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.094260 (+ 1175us) server_negotiation.cc:407] Connection header received
0927 10:53:47.095864 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.095867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.095932 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.096047 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.096954 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.097504 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.098648 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.098872 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.098983 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.100146 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.100150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.100153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:47.100172 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.100189 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:47.100197 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.100277 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.100433 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.100497 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:53:47.100953 6270 external_mini_cluster.cc:1428] Setting key eb07265dde5e74064431e5740f924de2
I20260927 10:53:47.101702 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.093442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:41815 (local address 127.6.31.129:40727)
0927 10:53:47.094095 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.094136 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.095649 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.096210 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.096217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.096296 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:47.096771 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.096782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.097648 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.097651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.098523 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.098532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.099052 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.099058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.099253 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.099899 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:47.099901 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:47.099938 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:47.099998 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:47.100279 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:47.100290 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:47.100298 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.100432 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.100769 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260927 10:53:47.103106 25120 heartbeater.cc:347] Connected to a master server at 127.6.31.190:41815
I20260927 10:53:47.103209 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:41815
--builtin_ntp_servers=127.6.31.148:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:53:47.103355 25120 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:47.103973 25120 heartbeater.cc:511] Master 127.6.31.190:41815 requested a full tablet report, sending...
I20260927 10:53:47.105783 24929 ts_manager.cc:194] Registered new tserver with Master: c824caf5762545fdbc6c75568a24e03c (127.6.31.129:41583)
I20260927 10:53:47.107159 24929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.129:40727
W20260927 10:53:47.216249 25123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:47.216533 25123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:47.216574 25123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:47.220779 25123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:47.220835 25123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:47.220891 25123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:53:47.225710 25123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:41815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:47.226843 25123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:47.228251 25123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:47.238284 25128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.238344 25123 server_base.cc:1034] running on GCE node
W20260927 10:53:47.238294 25129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.238294 25131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.238909 25123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:47.239609 25123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:47.240769 25123 hybrid_clock.cc:648] HybridClock initialized: now 1790506427240739 us; error 43 us; skew 500 ppm
I20260927 10:53:47.242599 25123 webserver.cc:460] Webserver started at http://127.6.31.130:41149/ using document root <none> and password file <none>
I20260927 10:53:47.243250 25123 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:47.243319 25123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:47.243546 25123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:47.245630 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "d56b983395964a38be0ff6fbb496ffc0"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-c60x"
server_key: "36cd73fbf9704863c873e5bf916a95d0"
server_key_iv: "41b064a5ed98dd2f428c82d97cf89bf8"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.246197 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "d56b983395964a38be0ff6fbb496ffc0"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-c60x"
server_key: "36cd73fbf9704863c873e5bf916a95d0"
server_key_iv: "41b064a5ed98dd2f428c82d97cf89bf8"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.250152 25123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:53:47.252733 25137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.254114 25123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:47.254271 25123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "d56b983395964a38be0ff6fbb496ffc0"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-c60x"
server_key: "36cd73fbf9704863c873e5bf916a95d0"
server_key_iv: "41b064a5ed98dd2f428c82d97cf89bf8"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.254402 25123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:47.266593 25123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:47.267417 25123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:47.267624 25123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:47.268249 25123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:47.269201 25123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:47.269279 25123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.269342 25123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:47.269393 25123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.278450 25123 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:36391
I20260927 10:53:47.278476 25250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:36391 every 8 connection(s)
I20260927 10:53:47.279422 25123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:47.280220 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 25123
I20260927 10:53:47.280357 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:47.280699 6270 external_mini_cluster.cc:1428] Setting key 1ce759d1d35a6249e259cf95bb40bffa
I20260927 10:53:47.283596 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:41815
--builtin_ntp_servers=127.6.31.148:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:53:47.290680 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.281328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:55099 (local address 127.6.31.190:41815)
0927 10:53:47.282854 (+ 1526us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.282858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.282880 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:53:47.284552 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.284558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.284630 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.284721 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.285817 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.286495 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.288377 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.288607 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.288701 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.290148 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.290151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.290153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:47.290165 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.290185 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:47.290193 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.290254 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.290499 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.290563 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1436}
I20260927 10:53:47.291958 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.281849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:41815 (local address 127.6.31.130:55099)
0927 10:53:47.282737 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.282799 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.284337 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.285055 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.285066 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.285170 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:47.285652 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.285665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.286635 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.286641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.288227 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.288241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.288808 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.288816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.289097 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.289800 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:47.289803 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:47.289866 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:47.289964 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:47.290284 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:47.290295 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:47.290304 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.290453 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.290796 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":148,"threads_started":1}
I20260927 10:53:47.293433 25251 heartbeater.cc:347] Connected to a master server at 127.6.31.190:41815
I20260927 10:53:47.293694 25251 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:47.294266 25251 heartbeater.cc:511] Master 127.6.31.190:41815 requested a full tablet report, sending...
I20260927 10:53:47.295533 24929 ts_manager.cc:194] Registered new tserver with Master: d56b983395964a38be0ff6fbb496ffc0 (127.6.31.130:36391)
I20260927 10:53:47.296244 24929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.130:55099
W20260927 10:53:47.400401 25254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:47.400851 25254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:47.401041 25254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:47.405469 25254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:47.405548 25254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:47.405664 25254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:53:47.410677 25254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:41815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:47.411830 25254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:47.413196 25254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:47.422708 25262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.422709 25260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.422755 25259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.423117 25254 server_base.cc:1034] running on GCE node
I20260927 10:53:47.423508 25254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:47.424141 25254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:47.425308 25254 hybrid_clock.cc:648] HybridClock initialized: now 1790506427425271 us; error 50 us; skew 500 ppm
I20260927 10:53:47.427211 25254 webserver.cc:460] Webserver started at http://127.6.31.131:36615/ using document root <none> and password file <none>
I20260927 10:53:47.427778 25254 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:47.427839 25254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:47.428114 25254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:47.430184 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "ff5f4ac2b910464d8cc08343e5e823d0"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-c60x"
server_key: "6385f0b5b887eb41c4806c6839c24c36"
server_key_iv: "9c08c67d9b158e500e3438b3a8c0b125"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.430728 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "ff5f4ac2b910464d8cc08343e5e823d0"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-c60x"
server_key: "6385f0b5b887eb41c4806c6839c24c36"
server_key_iv: "9c08c67d9b158e500e3438b3a8c0b125"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.434545 25254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 10:53:47.436920 25268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.438227 25254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:47.438385 25254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "ff5f4ac2b910464d8cc08343e5e823d0"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-c60x"
server_key: "6385f0b5b887eb41c4806c6839c24c36"
server_key_iv: "9c08c67d9b158e500e3438b3a8c0b125"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.438519 25254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:47.458407 25254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:47.459244 25254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:47.459451 25254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:47.460079 25254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:47.461086 25254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:47.461158 25254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.461217 25254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:47.461247 25254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.469815 25254 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:40093
I20260927 10:53:47.469836 25381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:40093 every 8 connection(s)
I20260927 10:53:47.470790 25254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:47.480377 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.472686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:58867 (local address 127.6.31.190:41815)
0927 10:53:47.472831 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.472836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.473935 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 10:53:47.475574 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.475577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.475640 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.475744 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.476632 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.477188 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.478333 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.478587 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.478702 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.479865 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.479868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.479871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:47.479885 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.479908 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:47.479922 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.480004 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.480165 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.480227 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:53:47.480867 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 25254
I20260927 10:53:47.480965 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:47.481248 6270 external_mini_cluster.cc:1428] Setting key 49afda9f92adc16beeaa464213e8661c
I20260927 10:53:47.481256 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.473094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:41815 (local address 127.6.31.131:58867)
0927 10:53:47.473789 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.473827 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.475334 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.475913 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.475920 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.476007 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:47.476480 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.476492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.477322 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.477325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.478200 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.478209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.478742 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.478746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.478910 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.479596 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:47.479597 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:47.479638 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:47.479703 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:47.480012 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:47.480022 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:47.480028 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.480165 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.480441 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":164,"threads_started":1}
I20260927 10:53:47.482518 25382 heartbeater.cc:347] Connected to a master server at 127.6.31.190:41815
I20260927 10:53:47.482745 25382 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:47.483320 25382 heartbeater.cc:511] Master 127.6.31.190:41815 requested a full tablet report, sending...
I20260927 10:53:47.484316 24932 ts_manager.cc:194] Registered new tserver with Master: ff5f4ac2b910464d8cc08343e5e823d0 (127.6.31.131:40093)
I20260927 10:53:47.484895 24932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.131:58867
I20260927 10:53:47.495595 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:47.501699 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.497704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.6.31.190:41815)
0927 10:53:47.497849 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.497853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.498048 (+ 195us) server_negotiation.cc:407] Connection header received
0927 10:53:47.498097 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.498099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.498151 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.498224 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.498536 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.499052 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.500067 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.500276 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.500354 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.501196 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.501199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.501201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:47.501213 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.501231 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:47.501239 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.501360 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.501474 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.501507 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:53:47.504683 24932 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:47.507798 24932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:47.522755 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.516701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:41583 (local address 127.0.0.1:40390)
0927 10:53:47.517076 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.517145 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.517359 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.518139 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.518147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.518173 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.518322 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.518330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.519763 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.519766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.521134 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.521143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.522212 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.522216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.522321 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.522361 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.522460 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.522578 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":51,"thread_start_us":80,"threads_started":1}
I20260927 10:53:47.522804 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.516850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.6.31.129:41583)
0927 10:53:47.517262 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.517273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.517301 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:53:47.517588 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.517595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.517803 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.517958 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.518456 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.519632 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.521268 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.522081 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.522202 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.522469 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.522524 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.522623 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":174,"threads_started":1}
I20260927 10:53:47.523193 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.517792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:40093 (local address 127.0.0.1:58596)
0927 10:53:47.518257 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.518272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.518352 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.518910 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.518913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.518939 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.519041 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.519048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.520189 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.520192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.521390 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.521401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.522724 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.522728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.522835 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.522860 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.522933 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.523015 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":68,"threads_started":1}
I20260927 10:53:47.523200 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.517482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:36391 (local address 127.0.0.1:45478)
0927 10:53:47.518053 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.518070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.518162 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.519039 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.519042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.519058 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.519201 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.519208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.520704 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.520708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.521828 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.521836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.522787 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.522792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.522901 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.522924 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.522975 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.523047 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260927 10:53:47.523617 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.518187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.6.31.131:40093)
0927 10:53:47.518571 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.518581 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.518600 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:53:47.518657 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.518660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.518780 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.518933 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.519198 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.520060 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.521612 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.522382 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.522506 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.522658 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.523358 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.523462 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":181,"threads_started":1}
I20260927 10:53:47.523795 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.517647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.6.31.130:36391)
0927 10:53:47.518595 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.518608 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.518630 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:53:47.518703 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.518708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.518875 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.519343 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.519478 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.520579 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.521949 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.522653 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.523290 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.523472 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.523533 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.523625 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":117,"threads_started":1}
I20260927 10:53:47.526289 25185 tablet_service.cc:1467] Processing CreateTablet for tablet 6c96e5a2d1564075bb9334cd5411690f (DEFAULT_TABLE table=test-table [id=b76c91026e2b417286a98a3576997eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:47.526212 25054 tablet_service.cc:1467] Processing CreateTablet for tablet 6c96e5a2d1564075bb9334cd5411690f (DEFAULT_TABLE table=test-table [id=b76c91026e2b417286a98a3576997eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:47.526209 25316 tablet_service.cc:1467] Processing CreateTablet for tablet 6c96e5a2d1564075bb9334cd5411690f (DEFAULT_TABLE table=test-table [id=b76c91026e2b417286a98a3576997eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:47.527299 25185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c96e5a2d1564075bb9334cd5411690f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.527305 25054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c96e5a2d1564075bb9334cd5411690f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.527305 25316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c96e5a2d1564075bb9334cd5411690f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.533389 25399 tablet_bootstrap.cc:492] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0: Bootstrap starting.
I20260927 10:53:47.534770 25400 tablet_bootstrap.cc:492] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c: Bootstrap starting.
I20260927 10:53:47.535228 25401 tablet_bootstrap.cc:492] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: Bootstrap starting.
I20260927 10:53:47.536877 25400 tablet_bootstrap.cc:654] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.537211 25401 tablet_bootstrap.cc:654] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.537698 25400 log.cc:824] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:47.537999 25401 log.cc:824] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:47.538044 25399 tablet_bootstrap.cc:654] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.538822 25399 log.cc:824] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:47.539894 25401 tablet_bootstrap.cc:492] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: No bootstrap required, opened a new log
I20260927 10:53:47.539947 25400 tablet_bootstrap.cc:492] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c: No bootstrap required, opened a new log
I20260927 10:53:47.540107 25401 ts_tablet_manager.cc:1397] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:53:47.540118 25400 ts_tablet_manager.cc:1397] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:53:47.540726 25399 tablet_bootstrap.cc:492] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0: No bootstrap required, opened a new log
I20260927 10:53:47.540897 25399 ts_tablet_manager.cc:1397] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260927 10:53:47.544013 25401 raft_consensus.cc:348] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } }
I20260927 10:53:47.544265 25401 raft_consensus.cc:374] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.544315 25401 raft_consensus.cc:729] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5f4ac2b910464d8cc08343e5e823d0, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.544556 25400 raft_consensus.cc:348] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } }
I20260927 10:53:47.544808 25400 raft_consensus.cc:374] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.544751 25401 consensus_queue.cc:260] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } }
I20260927 10:53:47.544866 25400 raft_consensus.cc:729] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c824caf5762545fdbc6c75568a24e03c, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.545254 25400 consensus_queue.cc:260] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } }
I20260927 10:53:47.545349 25399 raft_consensus.cc:348] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } }
I20260927 10:53:47.545594 25399 raft_consensus.cc:374] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.545666 25399 raft_consensus.cc:729] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56b983395964a38be0ff6fbb496ffc0, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.545802 25401 ts_tablet_manager.cc:1428] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:53:47.546161 25382 heartbeater.cc:503] Master 127.6.31.190:41815 was elected leader, sending a full tablet report...
I20260927 10:53:47.546099 25399 consensus_queue.cc:260] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } }
I20260927 10:53:47.546161 25120 heartbeater.cc:503] Master 127.6.31.190:41815 was elected leader, sending a full tablet report...
I20260927 10:53:47.546967 25399 ts_tablet_manager.cc:1428] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260927 10:53:47.547168 25251 heartbeater.cc:503] Master 127.6.31.190:41815 was elected leader, sending a full tablet report...
I20260927 10:53:47.546182 25400 ts_tablet_manager.cc:1428] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 10:53:47.592407 25121 tablet.cc:2369] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:47.721839 25383 tablet.cc:2369] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:47.723124 25407 raft_consensus.cc:482] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:47.723299 25407 raft_consensus.cc:504] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } }
I20260927 10:53:47.724368 25407 leader_election.cc:288] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c824caf5762545fdbc6c75568a24e03c (127.6.31.129:41583), ff5f4ac2b910464d8cc08343e5e823d0 (127.6.31.131:40093)
I20260927 10:53:47.729499 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.724831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:41339 (local address 127.6.31.129:41583)
0927 10:53:47.724982 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.724987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.725008 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:53:47.725079 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.725082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.725158 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.725273 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.725634 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.726561 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.728217 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.728839 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.728968 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.729017 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.729302 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.729371 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:53:47.729499 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.725253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:55727 (local address 127.6.31.131:40093)
0927 10:53:47.725396 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.725401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.725415 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:53:47.725473 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.725475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.725515 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.725625 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.725880 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.726681 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.728217 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.728839 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.728968 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.729017 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.729302 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.729371 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:53:47.729518 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.724656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:41583 (local address 127.6.31.130:41339)
0927 10:53:47.724793 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.724811 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.724916 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.725295 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.725300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.725326 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.725500 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.725506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.726677 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.726680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.728065 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.728077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.728983 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.728985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.729117 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.729179 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.729270 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.729357 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":104,"spinlock_wait_cycles":25472}
I20260927 10:53:47.729532 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.724949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:40093 (local address 127.6.31.130:55727)
0927 10:53:47.725279 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.725292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.725390 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.725648 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.725650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.725665 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.725767 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.725771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.726781 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.726784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.728065 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.728080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.728981 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.728985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.729158 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.729199 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.729270 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.729357 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":86,"spinlock_wait_cycles":896,"thread_start_us":121,"threads_started":1}
I20260927 10:53:47.730271 25074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c96e5a2d1564075bb9334cd5411690f" candidate_uuid: "d56b983395964a38be0ff6fbb496ffc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c824caf5762545fdbc6c75568a24e03c" is_pre_election: true
I20260927 10:53:47.730284 25336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c96e5a2d1564075bb9334cd5411690f" candidate_uuid: "d56b983395964a38be0ff6fbb496ffc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" is_pre_election: true
I20260927 10:53:47.730588 25074 raft_consensus.cc:2454] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56b983395964a38be0ff6fbb496ffc0 in term 0.
I20260927 10:53:47.730595 25336 raft_consensus.cc:2454] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56b983395964a38be0ff6fbb496ffc0 in term 0.
I20260927 10:53:47.731144 25141 leader_election.cc:302] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c824caf5762545fdbc6c75568a24e03c, d56b983395964a38be0ff6fbb496ffc0; no voters:
I20260927 10:53:47.731442 25407 raft_consensus.cc:2781] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:47.731519 25407 raft_consensus.cc:482] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:47.731549 25407 raft_consensus.cc:3037] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:47.732601 25407 raft_consensus.cc:504] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } }
I20260927 10:53:47.732956 25407 leader_election.cc:288] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [CANDIDATE]: Term 1 election: Requested vote from peers c824caf5762545fdbc6c75568a24e03c (127.6.31.129:41583), ff5f4ac2b910464d8cc08343e5e823d0 (127.6.31.131:40093)
I20260927 10:53:47.733358 25074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c96e5a2d1564075bb9334cd5411690f" candidate_uuid: "d56b983395964a38be0ff6fbb496ffc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c824caf5762545fdbc6c75568a24e03c"
I20260927 10:53:47.733408 25336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c96e5a2d1564075bb9334cd5411690f" candidate_uuid: "d56b983395964a38be0ff6fbb496ffc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5f4ac2b910464d8cc08343e5e823d0"
I20260927 10:53:47.733511 25074 raft_consensus.cc:3037] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:47.733531 25336 raft_consensus.cc:3037] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:47.734535 25336 raft_consensus.cc:2454] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56b983395964a38be0ff6fbb496ffc0 in term 1.
I20260927 10:53:47.734560 25074 raft_consensus.cc:2454] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56b983395964a38be0ff6fbb496ffc0 in term 1.
I20260927 10:53:47.734880 25141 leader_election.cc:302] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d56b983395964a38be0ff6fbb496ffc0, ff5f4ac2b910464d8cc08343e5e823d0; no voters:
I20260927 10:53:47.735064 25407 raft_consensus.cc:2781] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:47.735337 25407 raft_consensus.cc:686] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [term 1 LEADER]: Becoming Leader. State: Replica: d56b983395964a38be0ff6fbb496ffc0, State: Running, Role: LEADER
I20260927 10:53:47.735690 25407 consensus_queue.cc:237] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } }
I20260927 10:53:47.738962 24931 catalog_manager.cc:5495] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d56b983395964a38be0ff6fbb496ffc0 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "d56b983395964a38be0ff6fbb496ffc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d56b983395964a38be0ff6fbb496ffc0" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 36391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:47.763072 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.759395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.6.31.130:36391)
0927 10:53:47.759512 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.759515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.759581 (+ 66us) server_negotiation.cc:407] Connection header received
0927 10:53:47.759705 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.759708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.759765 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.759853 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:47.760168 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.760670 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.761542 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.761765 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.761851 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.762233 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:47.762660 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:47.762772 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.762925 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.762958 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 10:53:47.769456 25336 raft_consensus.cc:1261] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [term 1 FOLLOWER]: Refusing update from remote peer d56b983395964a38be0ff6fbb496ffc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:47.769456 25074 raft_consensus.cc:1261] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c [term 1 FOLLOWER]: Refusing update from remote peer d56b983395964a38be0ff6fbb496ffc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:47.770326 25409 consensus_queue.cc:1035] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5f4ac2b910464d8cc08343e5e823d0" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:47.770534 25407 consensus_queue.cc:1035] T 6c96e5a2d1564075bb9334cd5411690f P d56b983395964a38be0ff6fbb496ffc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c824caf5762545fdbc6c75568a24e03c" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 41583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:47.778719 25412 mvcc.cc:205] Tried to move back new op lower bound from 7333914328135540736 to 7333914328007979008. Current Snapshot: MvccSnapshot[applied={T|T < 7333914328135540736}]
I20260927 10:53:47.778801 25414 mvcc.cc:205] Tried to move back new op lower bound from 7333914328135540736 to 7333914328007979008. Current Snapshot: MvccSnapshot[applied={T|T < 7333914328135540736}]
I20260927 10:53:47.780211 25413 mvcc.cc:205] Tried to move back new op lower bound from 7333914328135540736 to 7333914328007979008. Current Snapshot: MvccSnapshot[applied={T|T < 7333914328135540736}]
I20260927 10:53:47.789705 24929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:47.789888 24929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:47.793656 24929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae11ffa1bee54bb5b4e11ccff4a8ff56: Sending DeleteTablet for 3 replicas of tablet 6c96e5a2d1564075bb9334cd5411690f
I20260927 10:53:47.794613 25054 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c96e5a2d1564075bb9334cd5411690f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:47 UTC) from {username='slave'} at 127.0.0.1:40390
I20260927 10:53:47.795290 25185 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c96e5a2d1564075bb9334cd5411690f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:47 UTC) from {username='slave'} at 127.0.0.1:45478
I20260927 10:53:47.795823 25316 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c96e5a2d1564075bb9334cd5411690f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:47 UTC) from {username='slave'} at 127.0.0.1:58596
I20260927 10:53:47.795912 25421 tablet_replica.cc:330] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c: stopping tablet replica
I20260927 10:53:47.795995 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 24992
I20260927 10:53:47.796273 25421 raft_consensus.cc:2229] T 6c96e5a2d1564075bb9334cd5411690f P c824caf5762545fdbc6c75568a24e03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:47.796938 25423 tablet_replica.cc:330] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: stopping tablet replica
I20260927 10:53:47.797204 25423 raft_consensus.cc:2229] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:47.797650 25423 raft_consensus.cc:2258] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:47.800597 25423 ts_tablet_manager.cc:1905] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:47.803571 25423 ts_tablet_manager.cc:1918] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:47.803680 25423 log.cc:1196] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/6c96e5a2d1564075bb9334cd5411690f
I20260927 10:53:47.803995 25423 ts_tablet_manager.cc:1939] T 6c96e5a2d1564075bb9334cd5411690f P ff5f4ac2b910464d8cc08343e5e823d0: Deleting consensus metadata
W20260927 10:53:47.805083 24919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:41583 (error 108)
I20260927 10:53:47.805747 24919 catalog_manager.cc:4841] TS ff5f4ac2b910464d8cc08343e5e823d0 (127.6.31.131:40093): tablet 6c96e5a2d1564075bb9334cd5411690f (table test-table [id=b76c91026e2b417286a98a3576997eb8]) successfully deleted
I20260927 10:53:47.806079 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 25123
I20260927 10:53:47.806434 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.806091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:41583 (local address 127.0.0.1:40398)
0927 10:53:47.806261 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.806347 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260927 10:53:47.806958 24919 catalog_manager.cc:4584] TS c824caf5762545fdbc6c75568a24e03c (127.6.31.129:41583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c96e5a2d1564075bb9334cd5411690f: Network error: Client connection negotiation failed: client connection to 127.6.31.129:41583: connect: Connection refused (error 111)
I20260927 10:53:47.814903 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 25254
I20260927 10:53:47.815204 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.814796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:36391 (local address 127.0.0.1:45498)
0927 10:53:47.814969 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.815048 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.130:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 10:53:47.822239 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 24901
2026-09-27T10:53:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:47 dist-test-slave-c60x krb5kdc[25427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:47 dist-test-slave-c60x krb5kdc[25427](info): set up 2 sockets
Sep 27 10:53:47 dist-test-slave-c60x krb5kdc[25427](info): commencing operation
krb5kdc: starting...
W20260927 10:53:48.875250 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:53:48 dist-test-slave-c60x krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:53:49.006850 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38869
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:38869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:49.122267 25440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:49.122601 25440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:49.122684 25440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:49.126889 25440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:49.127000 25440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:49.127039 25440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:49.127116 25440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:49.127175 25440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:49.132640 25440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:38869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:49.133762 25440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:49.134799 25440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:49.144750 25445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:49.144750 25446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:49.144752 25448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:49.144923 25440 server_base.cc:1034] running on GCE node
I20260927 10:53:49.146062 25440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:49.147544 25440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:49.148751 25440 hybrid_clock.cc:648] HybridClock initialized: now 1790506429148712 us; error 56 us; skew 500 ppm
Sep 27 10:53:49 dist-test-slave-c60x krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:49.152007 25440 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:53:49.153112 25440 webserver.cc:460] Webserver started at http://127.6.31.190:46059/ using document root <none> and password file <none>
I20260927 10:53:49.153715 25440 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:49.153812 25440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:49.154028 25440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:49.156253 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "7fc28e281cdc8bd8acf6bdf71bc81c91"
server_key_iv: "e901d290892daabceba720ed21344596"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.156869 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "7fc28e281cdc8bd8acf6bdf71bc81c91"
server_key_iv: "e901d290892daabceba720ed21344596"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.160878 25440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 10:53:49.163506 25455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.164826 25440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:53:49.164973 25440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "7fc28e281cdc8bd8acf6bdf71bc81c91"
server_key_iv: "e901d290892daabceba720ed21344596"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.165161 25440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:49.180244 25440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:49.183657 25440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:49.184015 25440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:49.192018 25440 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:38869
I20260927 10:53:49.192044 25507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:38869 every 8 connection(s)
I20260927 10:53:49.193143 25440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:49.193580 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 25440
I20260927 10:53:49.193732 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:49.194000 6270 external_mini_cluster.cc:1428] Setting key 55e8a40236f6a1f286dc97dd31e236bb
I20260927 10:53:49.196391 25508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:49.201928 25508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899: Bootstrap starting.
Sep 27 10:53:49 dist-test-slave-c60x krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:49.204334 25508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:49.205082 25508 log.cc:824] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:49.207219 25508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899: No bootstrap required, opened a new log
I20260927 10:53:49.208305 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.195507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.6.31.190:38869)
0927 10:53:49.195955 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:49.195969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:49.196053 (+ 84us) server_negotiation.cc:407] Connection header received
0927 10:53:49.196739 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:49.196756 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:49.197101 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:49.197437 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:49.198276 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.199319 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.200166 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.200657 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.200833 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.203441 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:49.203446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:49.203459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:49.203489 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:49.205407 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.205869 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.205877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.205889 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.205999 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.206290 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.206294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.206296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.206701 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:49.206898 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:49.207224 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 10:53:49.207552 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":151,"threads_started":1}
I20260927 10:53:49.210536 25508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38869 } }
I20260927 10:53:49.210814 25508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:49.210896 25508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f0d6f5f38e54cd3a6dc99fb335f0899, State: Initialized, Role: FOLLOWER
I20260927 10:53:49.211385 25508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38869 } }
I20260927 10:53:49.211541 25508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:49.211632 25508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:49.211732 25508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:49.212677 25508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38869 } }
I20260927 10:53:49.213027 25508 leader_election.cc:302] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f0d6f5f38e54cd3a6dc99fb335f0899; no voters:
I20260927 10:53:49.213342 25508 leader_election.cc:288] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:49.213516 25513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:49.213773 25513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [term 1 LEADER]: Becoming Leader. State: Replica: 7f0d6f5f38e54cd3a6dc99fb335f0899, State: Running, Role: LEADER
I20260927 10:53:49.214156 25513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38869 } }
I20260927 10:53:49.214636 25508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:49.215988 25514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38869 } } }
I20260927 10:53:49.215969 25515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f0d6f5f38e54cd3a6dc99fb335f0899. Latest consensus state: current_term: 1 leader_uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0d6f5f38e54cd3a6dc99fb335f0899" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38869 } } }
I20260927 10:53:49.216135 25514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:49.216375 25515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:49.216563 25523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:49.219969 25523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:49.225657 25523 catalog_manager.cc:1347] Generated new cluster ID: 6d8b5b8188fc430d811349d72360985e
I20260927 10:53:49.225739 25523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:49.235010 25523 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:49.235948 25523 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:49.243345 25523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899: Generated new TSK 0
I20260927 10:53:49.244029 25523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.31.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
Entry for principal kudu/127.6.31.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab.
I20260927 10:53:49.282543 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38869
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:49.394505 25534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:49.394804 25534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:49.394877 25534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:49.398906 25534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:49.398996 25534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:49.399106 25534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:53:49.404291 25534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.129.keytab
--principal=kudu/127.6.31.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:49.405287 25534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:49.406265 25534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:49.415282 25542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:49.415343 25540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:49.415558 25534 server_base.cc:1034] running on GCE node
W20260927 10:53:49.415570 25539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:49.416015 25534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:49.416695 25534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:49.417874 25534 hybrid_clock.cc:648] HybridClock initialized: now 1790506429417860 us; error 25 us; skew 500 ppm
Sep 27 10:53:49 dist-test-slave-c60x krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:49.420516 25534 init.cc:377] Logged in from keytab as kudu/127.6.31.129@KRBTEST.COM (short username kudu)
I20260927 10:53:49.421307 25534 webserver.cc:460] Webserver started at http://127.6.31.129:33507/ using document root <none> and password file <none>
I20260927 10:53:49.421854 25534 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:49.421942 25534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:49.422200 25534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:49.424217 25534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "9808b9715e6340479fb021cc34192d06"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "6844f30739f7e38805f3bb520c537315"
server_key_iv: "54895e388d8b6e76533e9276584289b3"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.424760 25534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "9808b9715e6340479fb021cc34192d06"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "6844f30739f7e38805f3bb520c537315"
server_key_iv: "54895e388d8b6e76533e9276584289b3"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.428267 25534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:49.430527 25549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.431740 25534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:49.431867 25534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "9808b9715e6340479fb021cc34192d06"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "6844f30739f7e38805f3bb520c537315"
server_key_iv: "54895e388d8b6e76533e9276584289b3"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.431975 25534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:49.450248 25534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:49.453460 25534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:49.453670 25534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:49.454267 25534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:49.455111 25534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:49.455184 25534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.455260 25534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:49.455307 25534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.464275 25534 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:42829
I20260927 10:53:49.464329 25662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:42829 every 8 connection(s)
I20260927 10:53:49.465179 25534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:49.469341 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 25534
I20260927 10:53:49.469473 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:49.469779 6270 external_mini_cluster.cc:1428] Setting key 426ed92d13ddc9a22fd991782679593f
Sep 27 10:53:49 dist-test-slave-c60x krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.129@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.130@KRBTEST.COM; defaulting to no policy
I20260927 10:53:49.480605 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.466925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:52861 (local address 127.6.31.190:38869)
0927 10:53:49.467114 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:49.467136 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:49.467953 (+ 817us) server_negotiation.cc:407] Connection header received
0927 10:53:49.468948 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:49.468951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:49.469010 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:49.469106 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:49.470583 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.471362 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.473115 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.473532 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.473663 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.476538 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:49.476544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:49.476549 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:49.476594 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:49.478302 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.478815 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.478820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.478823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.478914 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.479332 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.479336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.479339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.479615 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:49.479738 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:49.480264 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 10:53:49.480402 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:53:49.481518 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.467368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38869 (local address 127.6.31.129:52861)
0927 10:53:49.467812 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:49.467853 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:49.468744 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:49.469316 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:49.469325 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:49.469794 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:49.470427 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.470446 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.471484 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.471487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.472939 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.472955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.473692 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.473695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.473831 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.474683 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:49.474709 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:49.476362 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:49.478432 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:49.478437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:49.478449 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:49.478709 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:49.479046 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:49.479048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:49.479050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:49.479236 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:49.479751 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:49.479756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:49.480000 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 10:53:49.480208 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 10:53:49.480538 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.130@KRBTEST.COM" created.
I20260927 10:53:49.482803 25663 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38869
I20260927 10:53:49.483152 25663 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:49.483654 25663 heartbeater.cc:511] Master 127.6.31.190:38869 requested a full tablet report, sending...
I20260927 10:53:49.485139 25472 ts_manager.cc:194] Registered new tserver with Master: 9808b9715e6340479fb021cc34192d06 (127.6.31.129:42829)
I20260927 10:53:49.486251 25472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.129@KRBTEST.COM'} at 127.6.31.129:52861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
Entry for principal kudu/127.6.31.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab.
I20260927 10:53:49.498364 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38869
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:49.610194 25668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:49.610462 25668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:49.610507 25668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:49.614619 25668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:49.614686 25668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:49.614745 25668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:53:49.620208 25668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.130.keytab
--principal=kudu/127.6.31.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:49.621198 25668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:49.622143 25668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:49.631798 25674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:49.631973 25668 server_base.cc:1034] running on GCE node
W20260927 10:53:49.631798 25673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:49.631863 25676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:49.632432 25668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:49.633051 25668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:49.634208 25668 hybrid_clock.cc:648] HybridClock initialized: now 1790506429634195 us; error 28 us; skew 500 ppm
Sep 27 10:53:49 dist-test-slave-c60x krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:49.637048 25668 init.cc:377] Logged in from keytab as kudu/127.6.31.130@KRBTEST.COM (short username kudu)
I20260927 10:53:49.637903 25668 webserver.cc:460] Webserver started at http://127.6.31.130:33091/ using document root <none> and password file <none>
I20260927 10:53:49.638447 25668 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:49.638507 25668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:49.638715 25668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:49.640746 25668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "e3ae207badc147c4a2c243dc996ef602"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "b8b3fbd97b99d3200d05dbd7a1423b0c"
server_key_iv: "96b348e196115d5a62bf496331974151"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.641274 25668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "e3ae207badc147c4a2c243dc996ef602"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "b8b3fbd97b99d3200d05dbd7a1423b0c"
server_key_iv: "96b348e196115d5a62bf496331974151"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.644958 25668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 10:53:49.647318 25683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.648483 25668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 10:53:49.648622 25668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "e3ae207badc147c4a2c243dc996ef602"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "b8b3fbd97b99d3200d05dbd7a1423b0c"
server_key_iv: "96b348e196115d5a62bf496331974151"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.648730 25668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:49.663992 25668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:49.667137 25668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:49.667359 25668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:49.667984 25668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:49.668857 25668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:49.668939 25668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.669031 25668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:49.669082 25668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.678857 25668 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:42945
I20260927 10:53:49.678879 25796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:42945 every 8 connection(s)
I20260927 10:53:49.679943 25668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:49.684703 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 25668
I20260927 10:53:49.684808 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:49.685096 6270 external_mini_cluster.cc:1428] Setting key 9299d1f351b3f90a272ff1fd8b681126
Sep 27 10:53:49 dist-test-slave-c60x krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.130@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:49.693931 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.681874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:37755 (local address 127.6.31.190:38869)
0927 10:53:49.682062 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:49.682066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:49.682766 (+ 700us) server_negotiation.cc:407] Connection header received
0927 10:53:49.683694 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:49.683698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:49.683757 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:49.683859 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:49.685057 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.685731 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.687005 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.687299 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.687410 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.690018 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:49.690024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:49.690031 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:49.690064 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:49.691834 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.692358 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.692364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.692368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.692475 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.692793 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.692799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.692802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.693040 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:49.693156 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:49.693611 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 10:53:49.693742 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:53:49.694648 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.682175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38869 (local address 127.6.31.130:37755)
0927 10:53:49.682611 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:49.682657 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:49.683486 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:49.684031 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:49.684037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:49.684371 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:49.684899 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.684919 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.685852 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.685856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.686878 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.686887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.687443 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.687448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.687613 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.688224 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:49.688240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:49.689855 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:49.691953 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:49.691957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:49.691968 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:49.692220 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:49.692573 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:49.692575 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:49.692576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:49.692700 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:49.693150 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:49.693154 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:49.693373 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 10:53:49.693524 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:53:49.693768 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.6.31.131@KRBTEST.COM; defaulting to no policy
I20260927 10:53:49.695816 25797 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38869
I20260927 10:53:49.696148 25797 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:49.696592 25797 heartbeater.cc:511] Master 127.6.31.190:38869 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.131@KRBTEST.COM" created.
I20260927 10:53:49.697649 25472 ts_manager.cc:194] Registered new tserver with Master: e3ae207badc147c4a2c243dc996ef602 (127.6.31.130:42945)
I20260927 10:53:49.698463 25472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.130@KRBTEST.COM'} at 127.6.31.130:37755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
Entry for principal kudu/127.6.31.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab.
I20260927 10:53:49.714454 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38869
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:49.825416 25802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:49.825719 25802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:49.825804 25802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:49.830191 25802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:49.830281 25802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:49.830367 25802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:53:49.835829 25802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.131.keytab
--principal=kudu/127.6.31.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:38869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:49.836970 25802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:49.837968 25802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:49.847402 25807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:49.847455 25802 server_base.cc:1034] running on GCE node
W20260927 10:53:49.847369 25810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:49.847369 25808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:49.848131 25802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:49.848716 25802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:49.849938 25802 hybrid_clock.cc:648] HybridClock initialized: now 1790506429849910 us; error 38 us; skew 500 ppm
Sep 27 10:53:49 dist-test-slave-c60x krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:49.852836 25802 init.cc:377] Logged in from keytab as kudu/127.6.31.131@KRBTEST.COM (short username kudu)
I20260927 10:53:49.853724 25802 webserver.cc:460] Webserver started at http://127.6.31.131:39911/ using document root <none> and password file <none>
I20260927 10:53:49.854257 25802 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:49.854316 25802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:49.854472 25802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:49.856592 25802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "65f563617bc348b8b5c42a9fc9b2d629"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "2428bf3dfa14118fa470b46bd13eba2c"
server_key_iv: "22bcf46e55ea2aacd5028a391ba1989a"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.857141 25802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "65f563617bc348b8b5c42a9fc9b2d629"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "2428bf3dfa14118fa470b46bd13eba2c"
server_key_iv: "22bcf46e55ea2aacd5028a391ba1989a"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.860852 25802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 10:53:49.863181 25817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.864351 25802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:49.864440 25802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "65f563617bc348b8b5c42a9fc9b2d629"
format_stamp: "Formatted at 2026-09-27 10:53:49 on dist-test-slave-c60x"
server_key: "2428bf3dfa14118fa470b46bd13eba2c"
server_key_iv: "22bcf46e55ea2aacd5028a391ba1989a"
server_key_version: "encryptionkey@0"
I20260927 10:53:49.864521 25802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:49.886637 25802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:49.890010 25802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:49.890183 25802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:49.890810 25802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:49.891616 25802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:49.891664 25802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.891703 25802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:49.891719 25802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:49.900377 25802 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:37575
I20260927 10:53:49.900398 25930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:37575 every 8 connection(s)
I20260927 10:53:49.901326 25802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:53:49 dist-test-slave-c60x krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.131@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:53:49.911000 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 25802
I20260927 10:53:49.911185 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:49.911479 6270 external_mini_cluster.cc:1428] Setting key 0e029517d03e3ba58e5a9e41fb149006
I20260927 10:53:49.914896 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.903156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:48817 (local address 127.6.31.190:38869)
0927 10:53:49.903304 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:49.903308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:49.904104 (+ 796us) server_negotiation.cc:407] Connection header received
0927 10:53:49.905032 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:49.905036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:49.905096 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:49.905206 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:49.906349 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.906891 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.908146 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.908383 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.908494 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.910957 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:49.910965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:49.910968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:49.910993 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:49.912628 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.913222 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.913226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.913228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.913320 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.913706 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.913710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.913711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.913940 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:49.914052 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:49.914583 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 10:53:49.914730 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:49.915849 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.903498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38869 (local address 127.6.31.131:48817)
0927 10:53:49.903949 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:49.903993 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:49.904825 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:49.905379 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:49.905385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:49.905746 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:49.906204 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.906217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.907012 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.907016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.907968 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.907977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.908534 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.908538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.908716 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.909289 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:49.909306 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:49.910792 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:49.912766 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:49.912774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:49.912790 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:49.913128 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:49.913410 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:49.913412 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:49.913414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:49.913594 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:49.914048 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:49.914054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:49.914332 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 10:53:49.914537 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 10:53:49.914861 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260927 10:53:49.916939 25931 heartbeater.cc:347] Connected to a master server at 127.6.31.190:38869
I20260927 10:53:49.917150 25931 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:49.917569 25931 heartbeater.cc:511] Master 127.6.31.190:38869 requested a full tablet report, sending...
I20260927 10:53:49.918499 25472 ts_manager.cc:194] Registered new tserver with Master: 65f563617bc348b8b5c42a9fc9b2d629 (127.6.31.131:37575)
I20260927 10:53:49.919301 25472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.31.131@KRBTEST.COM'} at 127.6.31.131:48817
I20260927 10:53:49.925402 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:49.934793 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.927501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.6.31.190:38869)
0927 10:53:49.927629 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:49.927633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:49.927808 (+ 175us) server_negotiation.cc:407] Connection header received
0927 10:53:49.927852 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:49.927855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:49.927900 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:49.927974 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:49.928706 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.929228 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.930039 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.930334 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.930416 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.931868 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:49.931877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:49.931880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:49.931902 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:49.933132 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.933543 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.933548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.933550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.933630 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:49.933908 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:49.933911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:49.933913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:49.934136 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:49.934275 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:49.934509 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 10:53:49.934631 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:53:49.937860 25472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:49.941030 25472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:49.956009 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.949395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:37575 (local address 127.0.0.1:45568)
0927 10:53:49.950168 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:49.950224 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:49.950354 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:49.951202 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:49.951209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:49.951233 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:49.951389 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.951398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.952989 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.952993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.954511 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.954522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.955535 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.955539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.955643 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.955680 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 10:53:49.955798 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 10:53:49.955888 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":69,"thread_start_us":78,"threads_started":1}
I20260927 10:53:49.956015 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.950100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.6.31.131:37575)
0927 10:53:49.950697 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:49.950708 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:49.950734 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:53:49.950817 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:49.950822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:49.951037 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:49.951209 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:49.951527 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.952828 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.954633 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.955409 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.955517 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.955661 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:49.955822 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:53:49.955897 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":97,"threads_started":1}
I20260927 10:53:49.956620 25944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.949395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42829 (local address 127.0.0.1:48812)
0927 10:53:49.950470 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:49.950485 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:49.950577 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:49.951407 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:49.951411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:49.951440 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:49.951557 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.951564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.953032 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.953035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.954396 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.954405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.956198 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.956201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.956305 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.956331 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:53:49.956401 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:53:49.956471 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":219,"thread_start_us":95,"threads_started":1}
I20260927 10:53:49.956898 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.950214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.6.31.129:42829)
0927 10:53:49.950827 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:49.950849 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:49.950872 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:53:49.950950 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:49.950955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:49.951202 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:49.951348 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:49.951670 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.952900 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.954521 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.955404 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.955517 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.955661 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:49.956499 (+ 838us) server_negotiation.cc:299] Negotiation successful
0927 10:53:49.956593 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":104,"threads_started":1}
I20260927 10:53:49.957304 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.949827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.6.31.130:42945)
0927 10:53:49.950188 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:49.950200 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:49.950813 (+ 613us) server_negotiation.cc:407] Connection header received
0927 10:53:49.951050 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:49.951056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:49.951253 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:49.951363 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:49.951883 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.952795 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.954795 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.955416 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.955543 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.955688 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:49.957039 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0927 10:53:49.957155 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":128,"threads_started":1}
I20260927 10:53:49.957713 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:49.949547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:42945 (local address 127.0.0.1:42768)
0927 10:53:49.950721 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:49.950734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:49.950875 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:49.951569 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:49.951572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:49.951588 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:49.951734 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.951752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.952956 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.952962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.954686 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:49.954699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:49.956764 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:49.956767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:49.956867 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:49.956888 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:53:49.957514 (+ 626us) client_negotiation.cc:240] Negotiation successful
0927 10:53:49.957587 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":385,"thread_start_us":119,"threads_started":1}
I20260927 10:53:49.959192 25865 tablet_service.cc:1467] Processing CreateTablet for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4 (DEFAULT_TABLE table=test-table [id=b825cbc29259426da3a1c734ab29b21e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:49.959390 25597 tablet_service.cc:1467] Processing CreateTablet for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4 (DEFAULT_TABLE table=test-table [id=b825cbc29259426da3a1c734ab29b21e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:49.960326 25865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:49.960321 25597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:49.960203 25731 tablet_service.cc:1467] Processing CreateTablet for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4 (DEFAULT_TABLE table=test-table [id=b825cbc29259426da3a1c734ab29b21e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:49.961149 25731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:49.966539 25948 tablet_bootstrap.cc:492] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06: Bootstrap starting.
I20260927 10:53:49.968358 25950 tablet_bootstrap.cc:492] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602: Bootstrap starting.
I20260927 10:53:49.969012 25948 tablet_bootstrap.cc:654] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:49.970117 25948 log.cc:824] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:49.970287 25950 tablet_bootstrap.cc:654] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:49.971302 25950 log.cc:824] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:49.971442 25949 tablet_bootstrap.cc:492] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: Bootstrap starting.
I20260927 10:53:49.972205 25948 tablet_bootstrap.cc:492] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06: No bootstrap required, opened a new log
I20260927 10:53:49.972414 25948 ts_tablet_manager.cc:1397] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:53:49.973253 25950 tablet_bootstrap.cc:492] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602: No bootstrap required, opened a new log
I20260927 10:53:49.973420 25949 tablet_bootstrap.cc:654] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:49.973444 25950 ts_tablet_manager.cc:1397] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:53:49.974776 25949 log.cc:824] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:49.976035 25948 raft_consensus.cc:348] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } }
I20260927 10:53:49.976277 25948 raft_consensus.cc:374] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:49.976351 25948 raft_consensus.cc:729] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808b9715e6340479fb021cc34192d06, State: Initialized, Role: FOLLOWER
I20260927 10:53:49.976759 25949 tablet_bootstrap.cc:492] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: No bootstrap required, opened a new log
I20260927 10:53:49.976841 25948 consensus_queue.cc:260] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } }
I20260927 10:53:49.976965 25949 ts_tablet_manager.cc:1397] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260927 10:53:49.977620 25663 heartbeater.cc:503] Master 127.6.31.190:38869 was elected leader, sending a full tablet report...
I20260927 10:53:49.977933 25948 ts_tablet_manager.cc:1428] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:53:49.977244 25950 raft_consensus.cc:348] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } }
I20260927 10:53:49.978111 25950 raft_consensus.cc:374] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:49.978173 25950 raft_consensus.cc:729] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ae207badc147c4a2c243dc996ef602, State: Initialized, Role: FOLLOWER
I20260927 10:53:49.978612 25950 consensus_queue.cc:260] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } }
I20260927 10:53:49.979291 25797 heartbeater.cc:503] Master 127.6.31.190:38869 was elected leader, sending a full tablet report...
I20260927 10:53:49.979565 25950 ts_tablet_manager.cc:1428] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:53:49.980696 25949 raft_consensus.cc:348] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } }
I20260927 10:53:49.980923 25949 raft_consensus.cc:374] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:49.980983 25949 raft_consensus.cc:729] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f563617bc348b8b5c42a9fc9b2d629, State: Initialized, Role: FOLLOWER
I20260927 10:53:49.981402 25949 consensus_queue.cc:260] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } }
I20260927 10:53:49.982044 25931 heartbeater.cc:503] Master 127.6.31.190:38869 was elected leader, sending a full tablet report...
I20260927 10:53:49.982331 25949 ts_tablet_manager.cc:1428] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
W20260927 10:53:50.152344 25932 tablet.cc:2369] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:50.181198 25798 tablet.cc:2369] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:50.197185 25955 raft_consensus.cc:482] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:50.197402 25955 raft_consensus.cc:504] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } }
I20260927 10:53:50.198539 25955 leader_election.cc:288] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65f563617bc348b8b5c42a9fc9b2d629 (127.6.31.131:37575), 9808b9715e6340479fb021cc34192d06 (127.6.31.129:42829)
I20260927 10:53:50.203186 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.199210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:45985 (local address 127.6.31.131:37575)
0927 10:53:50.199329 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:50.199333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:50.199349 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:53:50.199468 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:50.199472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:50.199520 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:50.199603 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:50.199943 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.200709 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.202174 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.202567 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.202702 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.202758 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:50.202949 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 10:53:50.203029 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:53:50.203313 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.198782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42829 (local address 127.6.31.130:55771)
0927 10:53:50.199131 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:50.199148 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:50.199308 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:50.199698 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:50.199701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:50.199712 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:50.199847 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:50.199853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.200581 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.200587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:50.201992 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:50.202003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.202869 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.202875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:50.202980 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.202999 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:53:50.203075 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:53:50.203167 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":126,"thread_start_us":70,"threads_started":1}
I20260927 10:53:50.203316 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.198924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:55771 (local address 127.6.31.129:42829)
0927 10:53:50.199143 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:50.199149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:50.199220 (+ 71us) server_negotiation.cc:407] Connection header received
0927 10:53:50.199448 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:50.199452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:50.199504 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:50.199543 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:50.199975 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.200465 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.202139 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.202754 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.202871 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.202919 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:50.203118 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 10:53:50.203194 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 10:53:50.203917 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.198780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:37575 (local address 127.6.31.130:45985)
0927 10:53:50.199265 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:50.199276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:50.199353 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:50.199637 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:50.199643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:50.199666 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:50.199826 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:50.199834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.200820 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.200823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:50.202012 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:50.202018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.202692 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.202696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:50.202798 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.202829 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:53:50.203695 (+ 866us) client_negotiation.cc:240] Negotiation successful
0927 10:53:50.203776 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":134,"thread_start_us":72,"threads_started":1}
I20260927 10:53:50.203907 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2344c9131ede4c2f8ba4b7d7f4318ce4" candidate_uuid: "e3ae207badc147c4a2c243dc996ef602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808b9715e6340479fb021cc34192d06" is_pre_election: true
I20260927 10:53:50.204272 25617 raft_consensus.cc:2454] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ae207badc147c4a2c243dc996ef602 in term 0.
I20260927 10:53:50.204474 25885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2344c9131ede4c2f8ba4b7d7f4318ce4" candidate_uuid: "e3ae207badc147c4a2c243dc996ef602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65f563617bc348b8b5c42a9fc9b2d629" is_pre_election: true
I20260927 10:53:50.204777 25885 raft_consensus.cc:2454] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ae207badc147c4a2c243dc996ef602 in term 0.
I20260927 10:53:50.204823 25687 leader_election.cc:302] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808b9715e6340479fb021cc34192d06, e3ae207badc147c4a2c243dc996ef602; no voters:
I20260927 10:53:50.205127 25955 raft_consensus.cc:2781] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:50.205245 25955 raft_consensus.cc:482] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:50.205307 25955 raft_consensus.cc:3037] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:50.206328 25955 raft_consensus.cc:504] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } }
I20260927 10:53:50.206692 25955 leader_election.cc:288] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [CANDIDATE]: Term 1 election: Requested vote from peers 65f563617bc348b8b5c42a9fc9b2d629 (127.6.31.131:37575), 9808b9715e6340479fb021cc34192d06 (127.6.31.129:42829)
I20260927 10:53:50.207131 25885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2344c9131ede4c2f8ba4b7d7f4318ce4" candidate_uuid: "e3ae207badc147c4a2c243dc996ef602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65f563617bc348b8b5c42a9fc9b2d629"
I20260927 10:53:50.207141 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2344c9131ede4c2f8ba4b7d7f4318ce4" candidate_uuid: "e3ae207badc147c4a2c243dc996ef602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808b9715e6340479fb021cc34192d06"
I20260927 10:53:50.207279 25885 raft_consensus.cc:3037] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:50.207285 25617 raft_consensus.cc:3037] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:50.208338 25617 raft_consensus.cc:2454] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ae207badc147c4a2c243dc996ef602 in term 1.
I20260927 10:53:50.208505 25885 raft_consensus.cc:2454] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ae207badc147c4a2c243dc996ef602 in term 1.
I20260927 10:53:50.208630 25687 leader_election.cc:302] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808b9715e6340479fb021cc34192d06, e3ae207badc147c4a2c243dc996ef602; no voters:
I20260927 10:53:50.208807 25955 raft_consensus.cc:2781] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:50.209028 25955 raft_consensus.cc:686] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 1 LEADER]: Becoming Leader. State: Replica: e3ae207badc147c4a2c243dc996ef602, State: Running, Role: LEADER
I20260927 10:53:50.209400 25955 consensus_queue.cc:237] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } }
I20260927 10:53:50.212806 25471 catalog_manager.cc:5495] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3ae207badc147c4a2c243dc996ef602 (127.6.31.130). New cstate: current_term: 1 leader_uuid: "e3ae207badc147c4a2c243dc996ef602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3ae207badc147c4a2c243dc996ef602" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42945 } health_report { overall_health: HEALTHY } } }
W20260927 10:53:50.216506 25664 tablet.cc:2369] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:50.251456 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.248047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.6.31.130:42945)
0927 10:53:50.248233 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:50.248236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:50.248249 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:50.248324 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:50.248326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:50.248381 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:50.248476 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:50.248716 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.249206 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.250132 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.250331 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.250404 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.250664 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:50.251072 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:50.251185 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:50.251287 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 10:53:50.251347 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 10:53:50.257344 25617 raft_consensus.cc:1261] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [term 1 FOLLOWER]: Refusing update from remote peer e3ae207badc147c4a2c243dc996ef602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:50.257344 25885 raft_consensus.cc:1261] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [term 1 FOLLOWER]: Refusing update from remote peer e3ae207badc147c4a2c243dc996ef602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:50.258013 25959 consensus_queue.cc:1035] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65f563617bc348b8b5c42a9fc9b2d629" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:50.258208 25955 consensus_queue.cc:1035] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808b9715e6340479fb021cc34192d06" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 42829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:50.266031 25962 mvcc.cc:205] Tried to move back new op lower bound from 7333914338327130112 to 7333914338140426240. Current Snapshot: MvccSnapshot[applied={T|T < 7333914338327130112}]
I20260927 10:53:50.268028 25963 mvcc.cc:205] Tried to move back new op lower bound from 7333914338327130112 to 7333914338140426240. Current Snapshot: MvccSnapshot[applied={T|T < 7333914338327130112}]
I20260927 10:53:50.269874 25965 mvcc.cc:205] Tried to move back new op lower bound from 7333914338327130112 to 7333914338140426240. Current Snapshot: MvccSnapshot[applied={T|T < 7333914338327130112}]
I20260927 10:53:50.274752 25471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:50.274943 25471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:50.277285 25471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f0d6f5f38e54cd3a6dc99fb335f0899: Sending DeleteTablet for 3 replicas of tablet 2344c9131ede4c2f8ba4b7d7f4318ce4
I20260927 10:53:50.278079 25731 tablet_service.cc:1514] Processing DeleteTablet for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:50 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:42768
I20260927 10:53:50.278095 25865 tablet_service.cc:1514] Processing DeleteTablet for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:50 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:45568
I20260927 10:53:50.278607 25974 tablet_replica.cc:330] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: stopping tablet replica
I20260927 10:53:50.278653 25597 tablet_service.cc:1514] Processing DeleteTablet for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:50 UTC) from {username='kudu', principal='kudu/127.6.31.190@KRBTEST.COM'} at 127.0.0.1:48812
I20260927 10:53:50.278957 25974 raft_consensus.cc:2229] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:50.279222 25975 tablet_replica.cc:330] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06: stopping tablet replica
I20260927 10:53:50.279376 25974 raft_consensus.cc:2258] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:50.279554 25975 raft_consensus.cc:2229] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:50.279966 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 25534
I20260927 10:53:50.279971 25975 raft_consensus.cc:2258] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 9808b9715e6340479fb021cc34192d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:50.280989 25974 ts_tablet_manager.cc:1905] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:50.283238 25974 ts_tablet_manager.cc:1918] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:50.283360 25974 log.cc:1196] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/2344c9131ede4c2f8ba4b7d7f4318ce4
I20260927 10:53:50.283553 25973 tablet_replica.cc:330] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602: stopping tablet replica
I20260927 10:53:50.283802 25974 ts_tablet_manager.cc:1939] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P 65f563617bc348b8b5c42a9fc9b2d629: Deleting consensus metadata
I20260927 10:53:50.283952 25973 raft_consensus.cc:2229] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:50.284373 25973 raft_consensus.cc:2258] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:50.285374 25458 catalog_manager.cc:4841] TS 65f563617bc348b8b5c42a9fc9b2d629 (127.6.31.131:37575): tablet 2344c9131ede4c2f8ba4b7d7f4318ce4 (table test-table [id=b825cbc29259426da3a1c734ab29b21e]) successfully deleted
W20260927 10:53:50.288246 25459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:42829 (error 108)
I20260927 10:53:50.288339 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 25668
I20260927 10:53:50.288694 25973 ts_tablet_manager.cc:1905] T 2344c9131ede4c2f8ba4b7d7f4318ce4 P e3ae207badc147c4a2c243dc996ef602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:50.289799 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.289234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:42829 (local address 127.0.0.1:48822)
0927 10:53:50.289616 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:50.289693 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
W20260927 10:53:50.290241 25459 catalog_manager.cc:4584] TS 9808b9715e6340479fb021cc34192d06 (127.6.31.129:42829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2344c9131ede4c2f8ba4b7d7f4318ce4: Network error: Client connection negotiation failed: client connection to 127.6.31.129:42829: connect: Connection refused (error 111)
I20260927 10:53:50.294525 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 25802
I20260927 10:53:50.295665 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.294659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:42945 (local address 127.0.0.1:42792)
0927 10:53:50.295547 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:50.295600 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.130:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":818}
I20260927 10:53:50.300158 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 25440
2026-09-27T10:53:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2477 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T10:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:50Z Disabled control of system clock
I20260927 10:53:50.331219 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:44945
--webserver_interface=127.6.31.190
--webserver_port=0
--builtin_ntp_servers=127.6.31.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:53:50.442301 25980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:50.442585 25980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:50.442623 25980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:50.446704 25980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:50.446761 25980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:50.446777 25980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:50.446794 25980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:50.446810 25980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:50.451519 25980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:44945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:50.452623 25980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:50.453933 25980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:50.462814 25985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:50.462903 25980 server_base.cc:1034] running on GCE node
W20260927 10:53:50.462832 25986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:50.462814 25988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:50.464102 25980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:50.465731 25980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:50.466933 25980 hybrid_clock.cc:648] HybridClock initialized: now 1790506430466900 us; error 50 us; skew 500 ppm
I20260927 10:53:50.469543 25980 webserver.cc:460] Webserver started at http://127.6.31.190:40883/ using document root <none> and password file <none>
I20260927 10:53:50.470300 25980 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:50.470388 25980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:50.470652 25980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:50.473313 25980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "3dc01a9a74fd4a589097f566f3638bab"
format_stamp: "Formatted at 2026-09-27 10:53:50 on dist-test-slave-c60x"
server_key: "37bab54e458aba3bafcf86d4cffb2f78"
server_key_iv: "00163786bb02c8a7aeed08e4c11c1c3b"
server_key_version: "encryptionkey@0"
I20260927 10:53:50.473906 25980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "3dc01a9a74fd4a589097f566f3638bab"
format_stamp: "Formatted at 2026-09-27 10:53:50 on dist-test-slave-c60x"
server_key: "37bab54e458aba3bafcf86d4cffb2f78"
server_key_iv: "00163786bb02c8a7aeed08e4c11c1c3b"
server_key_version: "encryptionkey@0"
I20260927 10:53:50.477826 25980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:53:50.480329 25994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:50.481623 25980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:50.481771 25980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "3dc01a9a74fd4a589097f566f3638bab"
format_stamp: "Formatted at 2026-09-27 10:53:50 on dist-test-slave-c60x"
server_key: "37bab54e458aba3bafcf86d4cffb2f78"
server_key_iv: "00163786bb02c8a7aeed08e4c11c1c3b"
server_key_version: "encryptionkey@0"
I20260927 10:53:50.481899 25980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:50.511920 25980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:50.512697 25980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:50.512905 25980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:50.520438 25980 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:44945
I20260927 10:53:50.520459 26046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:44945 every 8 connection(s)
I20260927 10:53:50.521464 25980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:50.524652 26047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:50.527354 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 25980
I20260927 10:53:50.527498 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:50.527788 6270 external_mini_cluster.cc:1428] Setting key 1d909f646fa0901185e5acfee5d10552
I20260927 10:53:50.530452 26047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab: Bootstrap starting.
I20260927 10:53:50.532909 26047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:50.533766 26047 log.cc:824] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:50.535862 26047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab: No bootstrap required, opened a new log
I20260927 10:53:50.537518 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.529281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.6.31.190:44945)
0927 10:53:50.529899 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:50.529914 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:50.529949 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:53:50.530670 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:50.530689 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:50.531067 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:50.531401 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:50.531767 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.532695 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.533955 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.534447 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.534645 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.535632 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:50.535637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:50.535649 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:50.536137 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:50.536196 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:50.536216 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:50.536326 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:50.536479 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 10:53:50.536738 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":132,"threads_started":1}
I20260927 10:53:50.540220 26047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc01a9a74fd4a589097f566f3638bab" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44945 } }
I20260927 10:53:50.540441 26047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:50.540501 26047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc01a9a74fd4a589097f566f3638bab, State: Initialized, Role: FOLLOWER
I20260927 10:53:50.540917 26047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc01a9a74fd4a589097f566f3638bab" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44945 } }
I20260927 10:53:50.541044 26047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:50.541133 26047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:50.541255 26047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:50.542207 26047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc01a9a74fd4a589097f566f3638bab" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44945 } }
I20260927 10:53:50.542548 26047 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc01a9a74fd4a589097f566f3638bab; no voters:
I20260927 10:53:50.542855 26047 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:50.543002 26052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:50.543299 26052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [term 1 LEADER]: Becoming Leader. State: Replica: 3dc01a9a74fd4a589097f566f3638bab, State: Running, Role: LEADER
I20260927 10:53:50.543661 26052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc01a9a74fd4a589097f566f3638bab" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44945 } }
I20260927 10:53:50.544235 26047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:50.545296 26054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc01a9a74fd4a589097f566f3638bab. Latest consensus state: current_term: 1 leader_uuid: "3dc01a9a74fd4a589097f566f3638bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc01a9a74fd4a589097f566f3638bab" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44945 } } }
I20260927 10:53:50.545428 26054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:50.545917 26053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc01a9a74fd4a589097f566f3638bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc01a9a74fd4a589097f566f3638bab" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44945 } } }
I20260927 10:53:50.546007 26053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:50.546705 26061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:50.549373 26061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:50.555552 26061 catalog_manager.cc:1347] Generated new cluster ID: d5022bd8768546c8b53c31c870023c6f
I20260927 10:53:50.555634 26061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:50.571744 26061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:50.572660 26061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:50.585529 26061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab: Generated new TSK 0
I20260927 10:53:50.586202 26061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:50.591383 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.129:0
--local_ip_for_outbound_sockets=127.6.31.129
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44945
--builtin_ntp_servers=127.6.31.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:53:50.701695 26071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:50.701951 26071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:50.701989 26071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:50.706015 26071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:50.706069 26071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:50.706151 26071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.129
I20260927 10:53:50.710860 26071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.31.129
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.129
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:50.711954 26071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:50.713397 26071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:50.723114 26079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:50.723253 26077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:50.723114 26076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:50.723158 26071 server_base.cc:1034] running on GCE node
I20260927 10:53:50.723737 26071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:50.724397 26071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:50.725581 26071 hybrid_clock.cc:648] HybridClock initialized: now 1790506430725547 us; error 44 us; skew 500 ppm
I20260927 10:53:50.727626 26071 webserver.cc:460] Webserver started at http://127.6.31.129:40677/ using document root <none> and password file <none>
I20260927 10:53:50.728197 26071 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:50.728262 26071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:50.728469 26071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:50.730480 26071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/data/instance:
uuid: "54000f56f5584bf8a4d28b99de6356fc"
format_stamp: "Formatted at 2026-09-27 10:53:50 on dist-test-slave-c60x"
server_key: "b887e097f35d212d5a8989edeb5c4647"
server_key_iv: "3b6a63ddf67fe2f70c851457dda49f37"
server_key_version: "encryptionkey@0"
I20260927 10:53:50.731060 26071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance:
uuid: "54000f56f5584bf8a4d28b99de6356fc"
format_stamp: "Formatted at 2026-09-27 10:53:50 on dist-test-slave-c60x"
server_key: "b887e097f35d212d5a8989edeb5c4647"
server_key_iv: "3b6a63ddf67fe2f70c851457dda49f37"
server_key_version: "encryptionkey@0"
I20260927 10:53:50.734644 26071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:50.737293 26085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:50.738508 26071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:50.738651 26071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/wal
uuid: "54000f56f5584bf8a4d28b99de6356fc"
format_stamp: "Formatted at 2026-09-27 10:53:50 on dist-test-slave-c60x"
server_key: "b887e097f35d212d5a8989edeb5c4647"
server_key_iv: "3b6a63ddf67fe2f70c851457dda49f37"
server_key_version: "encryptionkey@0"
I20260927 10:53:50.738770 26071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:50.754223 26071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:50.754951 26071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:50.755208 26071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:50.755817 26071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:50.756759 26071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:50.756830 26071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:50.756901 26071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:50.756970 26071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:50.765818 26071 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.129:36029
I20260927 10:53:50.765846 26198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.129:36029 every 8 connection(s)
I20260927 10:53:50.766794 26071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:50.767632 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 26071
I20260927 10:53:50.767748 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:50.768105 6270 external_mini_cluster.cc:1428] Setting key 92adcabdd9770b0770a3a3c7c1766c6d
I20260927 10:53:50.770259 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.130:0
--local_ip_for_outbound_sockets=127.6.31.130
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44945
--builtin_ntp_servers=127.6.31.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:53:50.776654 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.768768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:56797 (local address 127.6.31.190:44945)
0927 10:53:50.768904 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:50.768908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:50.770066 (+ 1158us) server_negotiation.cc:407] Connection header received
0927 10:53:50.771757 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:50.771760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:50.771806 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:50.771878 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:50.772999 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.773534 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.774684 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.774918 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.775029 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.776161 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:50.776171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:50.776174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:50.776187 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:50.776205 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:50.776212 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:50.776288 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:50.776449 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:53:50.776507 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:50.777490 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.769196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44945 (local address 127.6.31.129:56797)
0927 10:53:50.769916 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:50.769963 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:50.771510 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:50.772111 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:50.772126 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:50.772242 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:50.772869 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:50.772883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.773672 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.773677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:50.774545 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:50.774563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.775070 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.775074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:50.775257 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.775912 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:50.775914 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:50.775952 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:50.776013 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:50.776301 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:50.776313 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:50.776319 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:53:50.776448 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 10:53:50.776728 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":149,"threads_started":1}
I20260927 10:53:50.778791 26199 heartbeater.cc:347] Connected to a master server at 127.6.31.190:44945
I20260927 10:53:50.779053 26199 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:50.779735 26199 heartbeater.cc:511] Master 127.6.31.190:44945 requested a full tablet report, sending...
I20260927 10:53:50.781322 26010 ts_manager.cc:194] Registered new tserver with Master: 54000f56f5584bf8a4d28b99de6356fc (127.6.31.129:36029)
I20260927 10:53:50.782512 26010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.129:56797
W20260927 10:53:50.882982 26202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:50.883342 26202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:50.883423 26202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:50.887813 26202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:50.887873 26202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:50.887930 26202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.130
I20260927 10:53:50.892899 26202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.31.130
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.130
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:50.894025 26202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:50.895473 26202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:50.904949 26208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:50.904937 26210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:50.904942 26207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:50.905105 26202 server_base.cc:1034] running on GCE node
I20260927 10:53:50.905637 26202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:50.906261 26202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:50.907441 26202 hybrid_clock.cc:648] HybridClock initialized: now 1790506430907418 us; error 34 us; skew 500 ppm
I20260927 10:53:50.909272 26202 webserver.cc:460] Webserver started at http://127.6.31.130:46713/ using document root <none> and password file <none>
I20260927 10:53:50.909837 26202 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:50.909926 26202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:50.910161 26202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:50.912211 26202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/data/instance:
uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e"
format_stamp: "Formatted at 2026-09-27 10:53:50 on dist-test-slave-c60x"
server_key: "1f6ad1ee70509ccba70a73e911d82484"
server_key_iv: "a1528de742f743d13ccaf8c1c75609dc"
server_key_version: "encryptionkey@0"
I20260927 10:53:50.912755 26202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance:
uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e"
format_stamp: "Formatted at 2026-09-27 10:53:50 on dist-test-slave-c60x"
server_key: "1f6ad1ee70509ccba70a73e911d82484"
server_key_iv: "a1528de742f743d13ccaf8c1c75609dc"
server_key_version: "encryptionkey@0"
I20260927 10:53:50.916343 26202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:50.918726 26216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:50.920001 26202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:50.920166 26202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/wal
uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e"
format_stamp: "Formatted at 2026-09-27 10:53:50 on dist-test-slave-c60x"
server_key: "1f6ad1ee70509ccba70a73e911d82484"
server_key_iv: "a1528de742f743d13ccaf8c1c75609dc"
server_key_version: "encryptionkey@0"
I20260927 10:53:50.920279 26202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:50.934337 26202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:50.935009 26202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:50.935256 26202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:50.935837 26202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:50.936750 26202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:50.936821 26202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:50.936892 26202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:50.936954 26202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:50.946621 26202 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.130:42215
I20260927 10:53:50.946645 26329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.130:42215 every 8 connection(s)
I20260927 10:53:50.947602 26202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:50.957093 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.949527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.130:53629 (local address 127.6.31.190:44945)
0927 10:53:50.949671 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:50.949675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:50.950627 (+ 952us) server_negotiation.cc:407] Connection header received
0927 10:53:50.952184 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:50.952188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:50.952252 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:50.952354 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:50.953266 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.953816 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.954966 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.955220 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.955354 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.956534 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:50.956537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:50.956545 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:50.956558 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:50.956578 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:50.956586 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:50.956659 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:50.956828 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 10:53:50.956895 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:53:50.957392 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 26202
I20260927 10:53:50.957530 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:50.957857 6270 external_mini_cluster.cc:1428] Setting key 3540fbc45a7ab6e18d2059c33bf20eae
I20260927 10:53:50.958246 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:50.949819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44945 (local address 127.6.31.130:53629)
0927 10:53:50.950451 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:50.950498 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:50.951947 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:50.952529 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:50.952536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:50.952616 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:50.953118 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:50.953132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.953948 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.953957 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:50.954836 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:50.954846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:50.955384 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:50.955389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:50.955652 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:50.956260 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:50.956262 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:50.956303 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:50.956370 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:50.956673 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:50.956684 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:50.956690 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:53:50.956827 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 10:53:50.957327 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
I20260927 10:53:50.959559 26330 heartbeater.cc:347] Connected to a master server at 127.6.31.190:44945
I20260927 10:53:50.959805 26330 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:50.960141 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.31.131:0
--local_ip_for_outbound_sockets=127.6.31.131
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44945
--builtin_ntp_servers=127.6.31.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:53:50.960459 26330 heartbeater.cc:511] Master 127.6.31.190:44945 requested a full tablet report, sending...
I20260927 10:53:50.961719 26010 ts_manager.cc:194] Registered new tserver with Master: d84e2d7d36fa4775b5f9f1e9ddfeb90e (127.6.31.130:42215)
I20260927 10:53:50.962297 26010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.130:53629
W20260927 10:53:51.071887 26333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:51.072193 26333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:51.072263 26333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:51.076259 26333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:51.076336 26333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:51.076426 26333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.31.131
I20260927 10:53:51.081180 26333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.31.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.31.131
--webserver_port=0
--tserver_master_addrs=127.6.31.190:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.31.131
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:51.082268 26333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:51.083725 26333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:51.093775 26338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:51.093873 26333 server_base.cc:1034] running on GCE node
W20260927 10:53:51.093775 26341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:51.093775 26339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:51.094471 26333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:51.095178 26333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:51.096393 26333 hybrid_clock.cc:648] HybridClock initialized: now 1790506431096381 us; error 32 us; skew 500 ppm
I20260927 10:53:51.098459 26333 webserver.cc:460] Webserver started at http://127.6.31.131:40987/ using document root <none> and password file <none>
I20260927 10:53:51.099171 26333 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:51.099275 26333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:51.099536 26333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:51.101727 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/data/instance:
uuid: "21963c1e374a45cfaa6fcd287a674c2d"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-c60x"
server_key: "492ea26c6b44f07ad9b6d08f685e3b47"
server_key_iv: "b7654752d7fd82288f4d142803b64df0"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.102375 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance:
uuid: "21963c1e374a45cfaa6fcd287a674c2d"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-c60x"
server_key: "492ea26c6b44f07ad9b6d08f685e3b47"
server_key_iv: "b7654752d7fd82288f4d142803b64df0"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.106387 26333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:53:51.108814 26347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.110003 26333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:51.110147 26333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/wal
uuid: "21963c1e374a45cfaa6fcd287a674c2d"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-c60x"
server_key: "492ea26c6b44f07ad9b6d08f685e3b47"
server_key_iv: "b7654752d7fd82288f4d142803b64df0"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.110267 26333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:51.141412 26333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:51.142160 26333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:51.142392 26333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:51.143021 26333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:51.144078 26333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:51.144160 26333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.144233 26333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:51.144277 26333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.153610 26333 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.131:43205
I20260927 10:53:51.153635 26460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.131:43205 every 8 connection(s)
I20260927 10:53:51.154569 26333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:51.157020 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 26333
I20260927 10:53:51.157121 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:51.157481 6270 external_mini_cluster.cc:1428] Setting key 63048846416eda50f39cfaa54274116d
I20260927 10:53:51.164461 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.156660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.131:52983 (local address 127.6.31.190:44945)
0927 10:53:51.156863 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.156868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.158045 (+ 1177us) server_negotiation.cc:407] Connection header received
0927 10:53:51.159861 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.159863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.159908 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.159976 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:51.160853 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.161373 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.162493 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.162707 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.162785 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.164057 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:51.164060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:51.164062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:51.164074 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:51.164092 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:51.164102 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:51.164161 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.164293 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.164343 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:53:51.165343 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.157034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:44945 (local address 127.6.31.131:52983)
0927 10:53:51.157887 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:51.157929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:51.159643 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:51.160176 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:51.160182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:51.160261 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:51.160736 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.160747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.161504 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.161506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.162388 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.162397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.162866 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.162870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.163033 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.163812 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:51.163814 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:51.163853 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:51.163915 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:51.164174 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:51.164180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:51.164183 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:53:51.164276 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:53:51.164491 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":151,"threads_started":1}
I20260927 10:53:51.166463 26461 heartbeater.cc:347] Connected to a master server at 127.6.31.190:44945
I20260927 10:53:51.166697 26461 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:51.167258 26461 heartbeater.cc:511] Master 127.6.31.190:44945 requested a full tablet report, sending...
I20260927 10:53:51.168251 26010 ts_manager.cc:194] Registered new tserver with Master: 21963c1e374a45cfaa6fcd287a674c2d (127.6.31.131:43205)
I20260927 10:53:51.168828 26010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.31.131:52983
I20260927 10:53:51.171420 6270 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:51.177006 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.173327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.6.31.190:44945)
0927 10:53:51.173448 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.173452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.173589 (+ 137us) server_negotiation.cc:407] Connection header received
0927 10:53:51.173689 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.173691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.173728 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.173793 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:51.174019 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.174510 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.175428 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.175641 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.175723 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.176576 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:51.176579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:51.176581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:51.176607 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:51.176624 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:51.176632 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:51.176700 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.176839 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.176888 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:53:51.179980 26010 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:51.183007 26010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:51.198735 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.192258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:42215 (local address 127.0.0.1:49994)
0927 10:53:51.192677 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:51.192693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:51.192780 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:51.194253 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:51.194256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:51.194272 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:51.194347 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.194353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.195377 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.195384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.197049 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.197060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.198029 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.198032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.198145 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.198262 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 10:53:51.198384 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 10:53:51.198545 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":184,"thread_start_us":112,"threads_started":1}
I20260927 10:53:51.198825 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.192939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.6.31.129:36029)
0927 10:53:51.193395 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.193406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.193428 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:53:51.193498 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.193504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.193675 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.194133 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:51.194262 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.195395 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.197058 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.197897 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.198024 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.198216 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.198542 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.198662 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":161,"threads_started":1}
I20260927 10:53:51.198735 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.192757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36029 (local address 127.0.0.1:43970)
0927 10:53:51.193118 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:51.193131 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:51.193220 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:51.193822 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:51.193825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:51.193842 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:51.193965 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.193970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.195518 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.195522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.196926 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.196936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.198089 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.198092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.198216 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.198263 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 10:53:51.198389 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 10:53:51.198544 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":162,"thread_start_us":89,"threads_started":1}
I20260927 10:53:51.199378 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.193407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.6.31.130:42215)
0927 10:53:51.193827 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.193839 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.193858 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:53:51.193926 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.193930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.194126 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.194270 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:51.194468 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.195267 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.197163 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.197897 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.198014 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.198216 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.199054 (+ 838us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.199199 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":143,"threads_started":1}
I20260927 10:53:51.200578 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.192092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:43205 (local address 127.0.0.1:42638)
0927 10:53:51.192546 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:51.192607 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:51.192756 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:51.193797 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:51.193804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:51.193827 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:51.193977 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.193984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.195492 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.195495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.197049 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.197060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.200158 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.200161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.200256 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.200278 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:53:51.200359 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:53:51.200429 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":229,"thread_start_us":78,"threads_started":1}
I20260927 10:53:51.200654 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.192264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.6.31.131:43205)
0927 10:53:51.193049 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.193061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.193088 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:53:51.193209 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.193215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.193423 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.193616 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:51.194117 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.195375 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.198292 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.199219 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.199912 (+ 693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.200102 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.200393 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.200497 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":123,"threads_started":1}
I20260927 10:53:51.202132 26133 tablet_service.cc:1467] Processing CreateTablet for tablet 82b0b1d83ab941c881484bf916bf29da (DEFAULT_TABLE table=test-table [id=e82fe87d18374ee08f01ff73ba47a6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:51.202150 26264 tablet_service.cc:1467] Processing CreateTablet for tablet 82b0b1d83ab941c881484bf916bf29da (DEFAULT_TABLE table=test-table [id=e82fe87d18374ee08f01ff73ba47a6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:51.203226 26133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b0b1d83ab941c881484bf916bf29da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:51.203233 26264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b0b1d83ab941c881484bf916bf29da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:51.203225 26395 tablet_service.cc:1467] Processing CreateTablet for tablet 82b0b1d83ab941c881484bf916bf29da (DEFAULT_TABLE table=test-table [id=e82fe87d18374ee08f01ff73ba47a6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:51.204278 26395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b0b1d83ab941c881484bf916bf29da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:51.209846 26478 tablet_bootstrap.cc:492] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc: Bootstrap starting.
I20260927 10:53:51.211421 26480 tablet_bootstrap.cc:492] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: Bootstrap starting.
I20260927 10:53:51.212325 26478 tablet_bootstrap.cc:654] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:51.213215 26478 log.cc:824] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:51.213656 26480 tablet_bootstrap.cc:654] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:51.214399 26479 tablet_bootstrap.cc:492] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: Bootstrap starting.
I20260927 10:53:51.214524 26480 log.cc:824] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:51.215942 26478 tablet_bootstrap.cc:492] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc: No bootstrap required, opened a new log
I20260927 10:53:51.216161 26478 ts_tablet_manager.cc:1397] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:53:51.216526 26479 tablet_bootstrap.cc:654] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:51.216514 26480 tablet_bootstrap.cc:492] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: No bootstrap required, opened a new log
I20260927 10:53:51.216751 26480 ts_tablet_manager.cc:1397] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:53:51.217265 26479 log.cc:824] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:51.218855 26479 tablet_bootstrap.cc:492] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: No bootstrap required, opened a new log
I20260927 10:53:51.219053 26479 ts_tablet_manager.cc:1397] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:53:51.220763 26478 raft_consensus.cc:348] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } }
I20260927 10:53:51.220852 26480 raft_consensus.cc:348] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } }
I20260927 10:53:51.221076 26480 raft_consensus.cc:374] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:51.221069 26478 raft_consensus.cc:374] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:51.221122 26480 raft_consensus.cc:729] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21963c1e374a45cfaa6fcd287a674c2d, State: Initialized, Role: FOLLOWER
I20260927 10:53:51.221122 26478 raft_consensus.cc:729] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54000f56f5584bf8a4d28b99de6356fc, State: Initialized, Role: FOLLOWER
I20260927 10:53:51.221592 26478 consensus_queue.cc:260] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } }
I20260927 10:53:51.221592 26480 consensus_queue.cc:260] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } }
I20260927 10:53:51.222645 26480 ts_tablet_manager.cc:1428] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:53:51.222645 26478 ts_tablet_manager.cc:1428] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:53:51.222653 26479 raft_consensus.cc:348] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } }
I20260927 10:53:51.222909 26479 raft_consensus.cc:374] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:51.222967 26479 raft_consensus.cc:729] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84e2d7d36fa4775b5f9f1e9ddfeb90e, State: Initialized, Role: FOLLOWER
I20260927 10:53:51.223008 26461 heartbeater.cc:503] Master 127.6.31.190:44945 was elected leader, sending a full tablet report...
I20260927 10:53:51.223027 26199 heartbeater.cc:503] Master 127.6.31.190:44945 was elected leader, sending a full tablet report...
I20260927 10:53:51.223397 26479 consensus_queue.cc:260] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } }
I20260927 10:53:51.224233 26330 heartbeater.cc:503] Master 127.6.31.190:44945 was elected leader, sending a full tablet report...
I20260927 10:53:51.224351 26479 ts_tablet_manager.cc:1428] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 10:53:51.267959 26200 tablet.cc:2369] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:51.405642 26462 tablet.cc:2369] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:51.448784 26331 tablet.cc:2369] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:51.463873 26485 raft_consensus.cc:482] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:51.464115 26485 raft_consensus.cc:504] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } }
I20260927 10:53:51.465318 26485 leader_election.cc:288] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21963c1e374a45cfaa6fcd287a674c2d (127.6.31.131:43205), d84e2d7d36fa4775b5f9f1e9ddfeb90e (127.6.31.130:42215)
I20260927 10:53:51.469286 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.465670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:60827 (local address 127.6.31.131:43205)
0927 10:53:51.465799 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.465804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.465953 (+ 149us) server_negotiation.cc:407] Connection header received
0927 10:53:51.466195 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.466198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.466239 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.466322 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:51.466583 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.467173 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.468271 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.468687 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.468855 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.468906 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.469093 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.469160 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:51.469303 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.465521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.131:43205 (local address 127.6.31.129:60827)
0927 10:53:51.465860 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:51.465872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:51.466068 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:51.466334 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:51.466336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:51.466352 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:51.466474 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.466480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.467282 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.467287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.468116 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.468124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.468856 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.468859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.468955 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.468989 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 10:53:51.469078 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:53:51.469148 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":88,"threads_started":1}
I20260927 10:53:51.469563 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.465748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.31.129:47903 (local address 127.6.31.130:42215)
0927 10:53:51.465887 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.465892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.466180 (+ 288us) server_negotiation.cc:407] Connection header received
0927 10:53:51.466356 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.466359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.466415 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.466508 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:51.466796 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.467401 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.468528 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.468950 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.469088 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.469136 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.469334 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.469430 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:51.469923 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.465704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.130:42215 (local address 127.6.31.129:47903)
0927 10:53:51.466050 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:51.466062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:51.466233 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:51.466518 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:51.466521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:51.466537 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:51.466676 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.466682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.467516 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.467523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.468400 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.468409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.469085 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.469090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.469202 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.469223 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:53:51.469316 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:53:51.469807 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260927 10:53:51.469918 26415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b0b1d83ab941c881484bf916bf29da" candidate_uuid: "54000f56f5584bf8a4d28b99de6356fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21963c1e374a45cfaa6fcd287a674c2d" is_pre_election: true
I20260927 10:53:51.470258 26415 raft_consensus.cc:2454] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54000f56f5584bf8a4d28b99de6356fc in term 0.
I20260927 10:53:51.470546 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b0b1d83ab941c881484bf916bf29da" candidate_uuid: "54000f56f5584bf8a4d28b99de6356fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" is_pre_election: true
I20260927 10:53:51.470814 26089 leader_election.cc:302] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21963c1e374a45cfaa6fcd287a674c2d, 54000f56f5584bf8a4d28b99de6356fc; no voters:
I20260927 10:53:51.470836 26284 raft_consensus.cc:2454] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54000f56f5584bf8a4d28b99de6356fc in term 0.
I20260927 10:53:51.471108 26485 raft_consensus.cc:2781] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:51.471220 26485 raft_consensus.cc:482] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:51.471295 26485 raft_consensus.cc:3037] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:51.472332 26485 raft_consensus.cc:504] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } }
I20260927 10:53:51.472736 26485 leader_election.cc:288] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [CANDIDATE]: Term 1 election: Requested vote from peers 21963c1e374a45cfaa6fcd287a674c2d (127.6.31.131:43205), d84e2d7d36fa4775b5f9f1e9ddfeb90e (127.6.31.130:42215)
I20260927 10:53:51.473115 26415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b0b1d83ab941c881484bf916bf29da" candidate_uuid: "54000f56f5584bf8a4d28b99de6356fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21963c1e374a45cfaa6fcd287a674c2d"
I20260927 10:53:51.473184 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b0b1d83ab941c881484bf916bf29da" candidate_uuid: "54000f56f5584bf8a4d28b99de6356fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e"
I20260927 10:53:51.473259 26415 raft_consensus.cc:3037] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:51.473305 26284 raft_consensus.cc:3037] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:51.474684 26415 raft_consensus.cc:2454] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54000f56f5584bf8a4d28b99de6356fc in term 1.
I20260927 10:53:51.474684 26284 raft_consensus.cc:2454] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54000f56f5584bf8a4d28b99de6356fc in term 1.
I20260927 10:53:51.475030 26086 leader_election.cc:302] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54000f56f5584bf8a4d28b99de6356fc, d84e2d7d36fa4775b5f9f1e9ddfeb90e; no voters:
I20260927 10:53:51.475275 26485 raft_consensus.cc:2781] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:51.475535 26485 raft_consensus.cc:686] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [term 1 LEADER]: Becoming Leader. State: Replica: 54000f56f5584bf8a4d28b99de6356fc, State: Running, Role: LEADER
I20260927 10:53:51.475895 26485 consensus_queue.cc:237] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } }
I20260927 10:53:51.479302 26009 catalog_manager.cc:5495] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 54000f56f5584bf8a4d28b99de6356fc (127.6.31.129). New cstate: current_term: 1 leader_uuid: "54000f56f5584bf8a4d28b99de6356fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54000f56f5584bf8a4d28b99de6356fc" member_type: VOTER last_known_addr { host: "127.6.31.129" port: 36029 } health_report { overall_health: HEALTHY } } }
I20260927 10:53:51.491974 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.487954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.6.31.129:36029)
0927 10:53:51.488114 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.488119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.488229 (+ 110us) server_negotiation.cc:407] Connection header received
0927 10:53:51.488437 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.488440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.488508 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.488630 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:51.488882 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.489425 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.490360 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.490617 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.490736 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.491065 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:51.491562 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:51.491699 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.491767 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.491822 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:53:51.497749 26415 raft_consensus.cc:1261] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [term 1 FOLLOWER]: Refusing update from remote peer 54000f56f5584bf8a4d28b99de6356fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:51.497749 26284 raft_consensus.cc:1261] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [term 1 FOLLOWER]: Refusing update from remote peer 54000f56f5584bf8a4d28b99de6356fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:51.498473 26485 consensus_queue.cc:1035] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84e2d7d36fa4775b5f9f1e9ddfeb90e" member_type: VOTER last_known_addr { host: "127.6.31.130" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:51.498682 26489 consensus_queue.cc:1035] T 82b0b1d83ab941c881484bf916bf29da P 54000f56f5584bf8a4d28b99de6356fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "21963c1e374a45cfaa6fcd287a674c2d" member_type: VOTER last_known_addr { host: "127.6.31.131" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:51.517305 26009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:51.517529 26009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:51.520536 26009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dc01a9a74fd4a589097f566f3638bab: Sending DeleteTablet for 3 replicas of tablet 82b0b1d83ab941c881484bf916bf29da
I20260927 10:53:51.521324 26395 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b0b1d83ab941c881484bf916bf29da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:51 UTC) from {username='slave'} at 127.0.0.1:42638
I20260927 10:53:51.521872 26264 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b0b1d83ab941c881484bf916bf29da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:51 UTC) from {username='slave'} at 127.0.0.1:49994
I20260927 10:53:51.521907 26133 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b0b1d83ab941c881484bf916bf29da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:51 UTC) from {username='slave'} at 127.0.0.1:43970
I20260927 10:53:51.522517 26502 tablet_replica.cc:330] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: stopping tablet replica
I20260927 10:53:51.522872 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 26071
I20260927 10:53:51.522891 26502 raft_consensus.cc:2229] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:51.523635 26502 raft_consensus.cc:2258] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:51.525357 26502 ts_tablet_manager.cc:1905] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:51.525502 26504 tablet_replica.cc:330] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: stopping tablet replica
I20260927 10:53:51.525817 26504 raft_consensus.cc:2229] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:51.526134 26504 raft_consensus.cc:2258] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:51.527631 26504 ts_tablet_manager.cc:1905] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:51.528118 26502 ts_tablet_manager.cc:1918] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:51.528220 26502 log.cc:1196] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-2/wal/wals/82b0b1d83ab941c881484bf916bf29da
I20260927 10:53:51.528527 26502 ts_tablet_manager.cc:1939] T 82b0b1d83ab941c881484bf916bf29da P 21963c1e374a45cfaa6fcd287a674c2d: Deleting consensus metadata
I20260927 10:53:51.529834 25998 catalog_manager.cc:4841] TS 21963c1e374a45cfaa6fcd287a674c2d (127.6.31.131:43205): tablet 82b0b1d83ab941c881484bf916bf29da (table test-table [id=e82fe87d18374ee08f01ff73ba47a6f8]) successfully deleted
I20260927 10:53:51.530527 26504 ts_tablet_manager.cc:1918] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:51.530637 26504 log.cc:1196] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: Deleting WAL directory at /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506175715859-6270-0/minicluster-data/ts-1/wal/wals/82b0b1d83ab941c881484bf916bf29da
I20260927 10:53:51.530901 26504 ts_tablet_manager.cc:1939] T 82b0b1d83ab941c881484bf916bf29da P d84e2d7d36fa4775b5f9f1e9ddfeb90e: Deleting consensus metadata
I20260927 10:53:51.531908 25995 catalog_manager.cc:4841] TS d84e2d7d36fa4775b5f9f1e9ddfeb90e (127.6.31.130:42215): tablet 82b0b1d83ab941c881484bf916bf29da (table test-table [id=e82fe87d18374ee08f01ff73ba47a6f8]) successfully deleted
W20260927 10:53:51.532541 25995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.31.129:36029 (error 108)
I20260927 10:53:51.532819 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 26202
I20260927 10:53:51.534852 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.534582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.129:36029 (local address 127.0.0.1:43986)
0927 10:53:51.534719 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:51.534786 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.129:36029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 10:53:51.534989 25995 catalog_manager.cc:4584] TS 54000f56f5584bf8a4d28b99de6356fc (127.6.31.129:36029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82b0b1d83ab941c881484bf916bf29da: Network error: Client connection negotiation failed: client connection to 127.6.31.129:36029: connect: Connection refused (error 111)
I20260927 10:53:51.540338 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 26333
I20260927 10:53:51.546020 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 25980
2026-09-27T10:53:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1244 ms)
[----------] 6 tests from EncryptionPolicyTest (10825 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:51 dist-test-slave-c60x krb5kdc[26508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:51 dist-test-slave-c60x krb5kdc[26508](info): set up 2 sockets
Sep 27 10:53:51 dist-test-slave-c60x krb5kdc[26508](info): commencing operation
krb5kdc: starting...
W20260927 10:53:52.583225 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:53:52 dist-test-slave-c60x krb5kdc[26508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:53:52.704341 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:52.811950 26521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:52.812209 26521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:52.812251 26521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:52.816135 26521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:52.816188 26521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:52.816207 26521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:52.816224 26521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:52.816241 26521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:52.821332 26521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:52.822319 26521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:52.823278 26521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:52.832037 26527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.832068 26526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.832032 26529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:52.832139 26521 server_base.cc:1034] running on GCE node
I20260927 10:53:52.832907 26521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:52.833953 26521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:52.835155 26521 hybrid_clock.cc:648] HybridClock initialized: now 1790506432835139 us; error 31 us; skew 500 ppm
Sep 27 10:53:52 dist-test-slave-c60x krb5kdc[26508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:52.842409 26521 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:53:52.843293 26521 webserver.cc:460] Webserver started at http://127.0.0.1:38071/ using document root <none> and password file <none>
I20260927 10:53:52.843798 26521 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:52.843855 26521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:52.844012 26521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:52.845932 26521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a"
format_stamp: "Formatted at 2026-09-27 10:53:52 on dist-test-slave-c60x"
server_key: "7cc862806e13cb4d45ad1bfde8046332"
server_key_iv: "886a7e6072ceb0f450b7f250893e9c86"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.846426 26521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a"
format_stamp: "Formatted at 2026-09-27 10:53:52 on dist-test-slave-c60x"
server_key: "7cc862806e13cb4d45ad1bfde8046332"
server_key_iv: "886a7e6072ceb0f450b7f250893e9c86"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.849995 26521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:53:52.852478 26536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.853674 26521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:52.853814 26521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a"
format_stamp: "Formatted at 2026-09-27 10:53:52 on dist-test-slave-c60x"
server_key: "7cc862806e13cb4d45ad1bfde8046332"
server_key_iv: "886a7e6072ceb0f450b7f250893e9c86"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.853927 26521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:52.873965 26521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:52.877365 26521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:52.877601 26521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:52.885083 26521 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45427
I20260927 10:53:52.885187 26588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45427 every 8 connection(s)
I20260927 10:53:52.886116 26521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:52.888933 26589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:52.890491 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 26521
I20260927 10:53:52.890645 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:52.890931 6270 external_mini_cluster.cc:1428] Setting key 56e248aa4439e1676f8731d7c22e4918
I20260927 10:53:52.894980 26589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a: Bootstrap starting.
I20260927 10:53:52.897285 26589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:52.898033 26589 log.cc:824] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:53:52 dist-test-slave-c60x krb5kdc[26508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:53:52.899974 26589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a: No bootstrap required, opened a new log
I20260927 10:53:52.904017 26589 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260927 10:53:52.904358 26589 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:52.904422 26589 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf69ae7c055e4a0aafe2e3d12d8fc86a, State: Initialized, Role: FOLLOWER
I20260927 10:53:52.904882 26589 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260927 10:53:52.905036 26589 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:52.905074 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.892440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33683 (local address 127.0.0.1:45427)
0927 10:53:52.892883 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.892898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.892933 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:53:52.893571 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.893587 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.893904 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.894229 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.895073 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.896126 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.896994 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.897523 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.897697 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.900354 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:52.900359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:52.900371 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:52.900397 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:52.902135 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:52.902644 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:52.902648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:52.902653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:52.902753 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:52.903121 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:52.903124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:52.903126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:52.903508 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:52.903752 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:52.904002 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 10:53:52.904216 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260927 10:53:52.905103 26589 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:52.905267 26589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:52.906414 26589 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260927 10:53:52.906765 26589 leader_election.cc:302] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf69ae7c055e4a0aafe2e3d12d8fc86a; no voters:
I20260927 10:53:52.907120 26589 leader_election.cc:288] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:52.907256 26594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:52.907464 26594 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [term 1 LEADER]: Becoming Leader. State: Replica: bf69ae7c055e4a0aafe2e3d12d8fc86a, State: Running, Role: LEADER
I20260927 10:53:52.907773 26594 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260927 10:53:52.908337 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:52.908432 26589 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:52.909556 26596 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf69ae7c055e4a0aafe2e3d12d8fc86a. Latest consensus state: current_term: 1 leader_uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } } }
I20260927 10:53:52.909685 26596 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:52.910817 26595 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf69ae7c055e4a0aafe2e3d12d8fc86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } } }
I20260927 10:53:52.911214 26595 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a [sys.catalog]: This master's current role is: LEADER
W20260927 10:53:52.913847 26613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:53:52.913955 26613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:53:52.914047 26606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:52.914880 26606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:52.919324 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.910810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32961 (local address 127.0.0.1:45427)
0927 10:53:52.910909 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.910914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.911435 (+ 521us) server_negotiation.cc:407] Connection header received
0927 10:53:52.911602 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.911605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.911649 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.911742 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.912387 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.912890 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.913710 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.913987 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.914089 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.915689 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:52.915693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:52.915696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:52.915719 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:52.917348 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:52.917906 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:52.917910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:52.917911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:52.917999 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:52.918364 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:52.918368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:52.918369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:52.918582 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:52.918727 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:52.919016 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 10:53:52.919163 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 10:53:52.920913 26606 catalog_manager.cc:1347] Generated new cluster ID: 21c018f640064dceab99a2d684de951e
I20260927 10:53:52.921060 26606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:52.935420 26606 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:52.936677 26606 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:52.949213 26606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf69ae7c055e4a0aafe2e3d12d8fc86a: Generated new TSK 0
I20260927 10:53:52.950057 26606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:52.957861 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 26521
W20260927 10:53:52.962735 26506 connection.cc:535] client connection to 127.0.0.1:45427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45427): Connection reset by peer (error 104)
2026-09-27T10:53:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:53 dist-test-slave-c60x krb5kdc[26620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:53 dist-test-slave-c60x krb5kdc[26620](info): set up 2 sockets
Sep 27 10:53:53 dist-test-slave-c60x krb5kdc[26620](info): commencing operation
krb5kdc: starting...
W20260927 10:53:53.983242 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:53:54 dist-test-slave-c60x krb5kdc[26620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:53:54.110050 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:54.222659 26633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:54.222967 26633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:54.223013 26633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:54.227298 26633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:54.227360 26633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:54.227380 26633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:54.227398 26633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:54.227449 26633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:54.232969 26633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:54.234061 26633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:54.235167 26633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:54.243865 26641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:54.243963 26639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:54.243865 26638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:54.244201 26633 server_base.cc:1034] running on GCE node
I20260927 10:53:54.244819 26633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:54.245919 26633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:54.247221 26633 hybrid_clock.cc:648] HybridClock initialized: now 1790506434247199 us; error 53 us; skew 500 ppm
Sep 27 10:53:54 dist-test-slave-c60x krb5kdc[26620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:54.250144 26633 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:53:54.251139 26633 webserver.cc:460] Webserver started at http://127.0.0.1:33193/ using document root <none> and password file <none>
I20260927 10:53:54.251730 26633 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:54.251801 26633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:54.252023 26633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:54.254149 26633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "9fa0b471f6514200acb7bed61602cdf1"
format_stamp: "Formatted at 2026-09-27 10:53:54 on dist-test-slave-c60x"
server_key: "359fc758785a8ca513d920a76483d00f"
server_key_iv: "d35fc7eed7dcc283c21885d0ae0a5f76"
server_key_version: "encryptionkey@0"
I20260927 10:53:54.254732 26633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "9fa0b471f6514200acb7bed61602cdf1"
format_stamp: "Formatted at 2026-09-27 10:53:54 on dist-test-slave-c60x"
server_key: "359fc758785a8ca513d920a76483d00f"
server_key_iv: "d35fc7eed7dcc283c21885d0ae0a5f76"
server_key_version: "encryptionkey@0"
I20260927 10:53:54.258710 26633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:53:54.261771 26648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:54.263226 26633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 10:53:54.263360 26633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "9fa0b471f6514200acb7bed61602cdf1"
format_stamp: "Formatted at 2026-09-27 10:53:54 on dist-test-slave-c60x"
server_key: "359fc758785a8ca513d920a76483d00f"
server_key_iv: "d35fc7eed7dcc283c21885d0ae0a5f76"
server_key_version: "encryptionkey@0"
I20260927 10:53:54.263525 26633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:54.280031 26633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:54.283545 26633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:54.283715 26633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:54.290972 26633 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37711
I20260927 10:53:54.290997 26700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37711 every 8 connection(s)
I20260927 10:53:54.291997 26633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:54.294786 26701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:54.297201 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 26633
I20260927 10:53:54.297354 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:54.297642 6270 external_mini_cluster.cc:1428] Setting key 1fb5ed725270a68f39f30a8d4ea9fa25
I20260927 10:53:54.300654 26701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1: Bootstrap starting.
I20260927 10:53:54.302767 26701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:54.303545 26701 log.cc:824] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:54.305343 26701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1: No bootstrap required, opened a new log
I20260927 10:53:54.306430 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:54.299048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41143 (local address 127.0.0.1:37711)
0927 10:53:54.299502 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:54.299517 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:54.299552 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:53:54.300186 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:54.300203 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:54.300528 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:54.300866 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:54.301291 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.302255 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.303253 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.303782 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.303949 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:54.304837 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:54.304848 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:54.304860 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:54.305241 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:54.305291 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:54.305313 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:54.305452 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:54.305606 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 10:53:54.305754 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260927 10:53:54.309501 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:54.309443 26701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa0b471f6514200acb7bed61602cdf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37711 } }
I20260927 10:53:54.309726 26701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:54.309783 26701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa0b471f6514200acb7bed61602cdf1, State: Initialized, Role: FOLLOWER
I20260927 10:53:54.310196 26701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa0b471f6514200acb7bed61602cdf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37711 } }
I20260927 10:53:54.310343 26701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:54.310406 26701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:54.310496 26701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:54.311626 26701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa0b471f6514200acb7bed61602cdf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37711 } }
I20260927 10:53:54.311981 26701 leader_election.cc:302] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa0b471f6514200acb7bed61602cdf1; no voters:
I20260927 10:53:54.312302 26701 leader_election.cc:288] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:54.312389 26711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:54.312665 26711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [term 1 LEADER]: Becoming Leader. State: Replica: 9fa0b471f6514200acb7bed61602cdf1, State: Running, Role: LEADER
I20260927 10:53:54.313050 26711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa0b471f6514200acb7bed61602cdf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37711 } }
I20260927 10:53:54.313592 26701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:54.314687 26712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa0b471f6514200acb7bed61602cdf1. Latest consensus state: current_term: 1 leader_uuid: "9fa0b471f6514200acb7bed61602cdf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa0b471f6514200acb7bed61602cdf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37711 } } }
I20260927 10:53:54.314837 26712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:54.315145 26713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa0b471f6514200acb7bed61602cdf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa0b471f6514200acb7bed61602cdf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37711 } } }
I20260927 10:53:54.315237 26713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:54.315442 26720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:54.318197 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:54.311813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55571 (local address 127.0.0.1:37711)
0927 10:53:54.311919 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:54.311923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:54.312174 (+ 251us) server_negotiation.cc:407] Connection header received
0927 10:53:54.312336 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:54.312340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:54.312389 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:54.312475 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:54.312895 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.313625 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.314802 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.316250 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.316388 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:54.317686 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:54.317690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:54.317693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:54.317706 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:54.317736 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:54.317750 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:54.317848 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:54.318017 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 10:53:54.318053 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":30}
I20260927 10:53:54.319116 26720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:54.324800 26720 catalog_manager.cc:1347] Generated new cluster ID: 411430e388e34f90b544b6edbfda4082
I20260927 10:53:54.324877 26720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:54.334367 26720 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:54.335371 26720 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:54.340405 26720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fa0b471f6514200acb7bed61602cdf1: Generated new TSK 0
I20260927 10:53:54.341056 26720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:54.345249 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 26633
W20260927 10:53:54.350672 26618 connection.cc:535] client connection to 127.0.0.1:37711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37711): Connection reset by peer (error 104)
2026-09-27T10:53:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:54 dist-test-slave-c60x krb5kdc[26732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:54 dist-test-slave-c60x krb5kdc[26732](info): set up 2 sockets
Sep 27 10:53:54 dist-test-slave-c60x krb5kdc[26732](info): commencing operation
krb5kdc: starting...
W20260927 10:53:55.399410 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:53:55 dist-test-slave-c60x krb5kdc[26732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:53:55.521799 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:55.630069 26745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:55.630379 26745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:55.630455 26745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:55.634503 26745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:55.634601 26745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:55.634639 26745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:55.634706 26745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:55.634745 26745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:55.640097 26745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:55.641206 26745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:55.642246 26745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:55.651270 26745 server_base.cc:1034] running on GCE node
W20260927 10:53:55.651425 26753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:55.651461 26751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:55.651641 26750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:55.652202 26745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:55.653352 26745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:55.654547 26745 hybrid_clock.cc:648] HybridClock initialized: now 1790506435654530 us; error 33 us; skew 500 ppm
Sep 27 10:53:55 dist-test-slave-c60x krb5kdc[26732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:55.657416 26745 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:53:55.658284 26745 webserver.cc:460] Webserver started at http://127.0.0.1:43079/ using document root <none> and password file <none>
I20260927 10:53:55.658805 26745 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:55.658860 26745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:55.659025 26745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:55.661096 26745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "e854a1518857441f8938e87f6f24479b"
format_stamp: "Formatted at 2026-09-27 10:53:55 on dist-test-slave-c60x"
server_key: "9f1fbaafcc74daa144899b6a128486e7"
server_key_iv: "5ef2ac74a575d0010b8a71c3e8170c26"
server_key_version: "encryptionkey@0"
I20260927 10:53:55.661610 26745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "e854a1518857441f8938e87f6f24479b"
format_stamp: "Formatted at 2026-09-27 10:53:55 on dist-test-slave-c60x"
server_key: "9f1fbaafcc74daa144899b6a128486e7"
server_key_iv: "5ef2ac74a575d0010b8a71c3e8170c26"
server_key_version: "encryptionkey@0"
I20260927 10:53:55.665445 26745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 10:53:55.668057 26760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:55.669304 26745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:55.669427 26745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "e854a1518857441f8938e87f6f24479b"
format_stamp: "Formatted at 2026-09-27 10:53:55 on dist-test-slave-c60x"
server_key: "9f1fbaafcc74daa144899b6a128486e7"
server_key_iv: "5ef2ac74a575d0010b8a71c3e8170c26"
server_key_version: "encryptionkey@0"
I20260927 10:53:55.669539 26745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:55.674099 26745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:55.677405 26745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:55.677597 26745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:55.684046 26745 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39577
I20260927 10:53:55.684108 26812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39577 every 8 connection(s)
I20260927 10:53:55.684965 26745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:55.687765 26813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:55.687757 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 26745
I20260927 10:53:55.687924 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:55.688205 6270 external_mini_cluster.cc:1428] Setting key b5359085e65ef08b6ea3b14038aeaccd
I20260927 10:53:55.694108 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:55.689590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58149 (local address 127.0.0.1:39577)
0927 10:53:55.689990 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:55.690007 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:55.690052 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:53:55.691146 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:55.691171 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:55.691517 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:55.691887 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:55.692602 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:55.692608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:55.692621 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:55.692632 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:55.692704 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:55.692729 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:55.692868 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:55.692935 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 10:53:55.693230 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260927 10:53:55.694904 26813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b: Bootstrap starting.
I20260927 10:53:55.696774 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:55.697335 26813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:55.698105 26813 log.cc:824] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:55.700361 26813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b: No bootstrap required, opened a new log
I20260927 10:53:55.700471 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:55.698860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38273 (local address 127.0.0.1:39577)
0927 10:53:55.698971 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:55.698975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:55.699129 (+ 154us) server_negotiation.cc:407] Connection header received
0927 10:53:55.699261 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:55.699263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:55.699290 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:55.699352 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:55.700051 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:55.700053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:55.700055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:55.700066 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:55.700088 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:55.700097 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:55.700163 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:55.700281 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 10:53:55.700310 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:53:55.703922 26813 raft_consensus.cc:348] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a1518857441f8938e87f6f24479b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39577 } }
I20260927 10:53:55.704177 26813 raft_consensus.cc:374] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:55.704257 26813 raft_consensus.cc:729] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e854a1518857441f8938e87f6f24479b, State: Initialized, Role: FOLLOWER
I20260927 10:53:55.704660 26813 consensus_queue.cc:260] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a1518857441f8938e87f6f24479b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39577 } }
I20260927 10:53:55.704818 26813 raft_consensus.cc:388] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:55.704918 26813 raft_consensus.cc:482] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:55.705025 26813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:55.705974 26813 raft_consensus.cc:504] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a1518857441f8938e87f6f24479b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39577 } }
I20260927 10:53:55.706364 26813 leader_election.cc:302] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e854a1518857441f8938e87f6f24479b; no voters:
I20260927 10:53:55.706723 26813 leader_election.cc:288] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:55.706877 26823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:55.707252 26823 raft_consensus.cc:686] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [term 1 LEADER]: Becoming Leader. State: Replica: e854a1518857441f8938e87f6f24479b, State: Running, Role: LEADER
I20260927 10:53:55.707629 26823 consensus_queue.cc:237] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a1518857441f8938e87f6f24479b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39577 } }
I20260927 10:53:55.708019 26813 sys_catalog.cc:564] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:55.709781 26824 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e854a1518857441f8938e87f6f24479b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a1518857441f8938e87f6f24479b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39577 } } }
I20260927 10:53:55.709921 26824 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:55.709995 26825 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e854a1518857441f8938e87f6f24479b. Latest consensus state: current_term: 1 leader_uuid: "e854a1518857441f8938e87f6f24479b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a1518857441f8938e87f6f24479b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39577 } } }
I20260927 10:53:55.710389 26825 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b [sys.catalog]: This master's current role is: LEADER
W20260927 10:53:55.712518 26837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:53:55.712582 26837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:53:55.712677 26838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:55.713369 26838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:55.719060 26838 catalog_manager.cc:1347] Generated new cluster ID: 926e1c01991a4845b65b6feef1f2e17d
I20260927 10:53:55.719174 26838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:55.740691 26838 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:55.741122 26838 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:55.746246 26838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e854a1518857441f8938e87f6f24479b: Generated new TSK 0
I20260927 10:53:55.747021 26838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:55.755218 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 26745
2026-09-27T10:53:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:55 dist-test-slave-c60x krb5kdc[26844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:55 dist-test-slave-c60x krb5kdc[26844](info): set up 2 sockets
Sep 27 10:53:55 dist-test-slave-c60x krb5kdc[26844](info): commencing operation
krb5kdc: starting...
W20260927 10:53:56.803381 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:53:56 dist-test-slave-c60x krb5kdc[26844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:53:56.930790 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:57.040331 26857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:57.040628 26857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:57.040722 26857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:57.044747 26857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:57.044800 26857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:57.044821 26857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:57.044836 26857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:57.044902 26857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:57.050225 26857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:57.051340 26857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:57.052474 26857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:57.061507 26863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:57.061582 26857 server_base.cc:1034] running on GCE node
W20260927 10:53:57.061512 26862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:57.061512 26865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:57.062517 26857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:57.063747 26857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:57.064952 26857 hybrid_clock.cc:648] HybridClock initialized: now 1790506437064898 us; error 68 us; skew 500 ppm
Sep 27 10:53:57 dist-test-slave-c60x krb5kdc[26844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:57.068076 26857 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:53:57.069155 26857 webserver.cc:460] Webserver started at http://127.0.0.1:33001/ using document root <none> and password file <none>
I20260927 10:53:57.069756 26857 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:57.069854 26857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:57.070094 26857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:57.072417 26857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "472192d5416f45c5a1b1eee94531c363"
format_stamp: "Formatted at 2026-09-27 10:53:57 on dist-test-slave-c60x"
server_key: "121ee90309da0dc665d7ccf67f3d8cef"
server_key_iv: "0df24479280d2365ff2105b9f7ebacbb"
server_key_version: "encryptionkey@0"
I20260927 10:53:57.072999 26857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "472192d5416f45c5a1b1eee94531c363"
format_stamp: "Formatted at 2026-09-27 10:53:57 on dist-test-slave-c60x"
server_key: "121ee90309da0dc665d7ccf67f3d8cef"
server_key_iv: "0df24479280d2365ff2105b9f7ebacbb"
server_key_version: "encryptionkey@0"
I20260927 10:53:57.077216 26857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:53:57.079819 26872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:57.081118 26857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:57.081265 26857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "472192d5416f45c5a1b1eee94531c363"
format_stamp: "Formatted at 2026-09-27 10:53:57 on dist-test-slave-c60x"
server_key: "121ee90309da0dc665d7ccf67f3d8cef"
server_key_iv: "0df24479280d2365ff2105b9f7ebacbb"
server_key_version: "encryptionkey@0"
I20260927 10:53:57.081389 26857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:57.099516 26857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:57.102883 26857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:57.103116 26857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:57.110353 26857 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44513
I20260927 10:53:57.110370 26924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44513 every 8 connection(s)
I20260927 10:53:57.111332 26857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:57.113971 26925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:57.117482 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 26857
I20260927 10:53:57.117641 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:57.117930 6270 external_mini_cluster.cc:1428] Setting key 3834c32923f027ec4ffde6dc5517a6c5
I20260927 10:53:57.119642 26925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363: Bootstrap starting.
I20260927 10:53:57.121780 26925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:57.122500 26925 log.cc:824] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:57.124395 26925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363: No bootstrap required, opened a new log
Sep 27 10:53:57 dist-test-slave-c60x krb5kdc[26844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:53:57.128559 26925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472192d5416f45c5a1b1eee94531c363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } }
I20260927 10:53:57.128841 26925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:57.128911 26925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472192d5416f45c5a1b1eee94531c363, State: Initialized, Role: FOLLOWER
I20260927 10:53:57.129395 26925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472192d5416f45c5a1b1eee94531c363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } }
I20260927 10:53:57.129550 26925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:57.129633 26925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:57.129750 26925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:57.130939 26925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472192d5416f45c5a1b1eee94531c363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } }
I20260927 10:53:57.131325 26925 leader_election.cc:302] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472192d5416f45c5a1b1eee94531c363; no voters:
I20260927 10:53:57.131685 26925 leader_election.cc:288] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:57.131843 26930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:57.132333 26930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [term 1 LEADER]: Becoming Leader. State: Replica: 472192d5416f45c5a1b1eee94531c363, State: Running, Role: LEADER
I20260927 10:53:57.132691 26930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472192d5416f45c5a1b1eee94531c363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } }
I20260927 10:53:57.133008 26925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:57.133261 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:57.119336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50355 (local address 127.0.0.1:44513)
0927 10:53:57.119828 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:57.119845 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:57.119881 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:53:57.120534 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:57.120550 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:57.120881 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:57.121223 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:57.122036 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.123248 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.124110 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.124641 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.124813 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:57.127762 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:57.127768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:57.127782 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:57.127809 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:57.129947 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:57.130421 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:57.130426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:57.130431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:57.130540 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:57.130832 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:57.130834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:57.130835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:57.131342 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:57.131571 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:57.131938 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 10:53:57.132276 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":141,"threads_started":1}
I20260927 10:53:57.133872 26930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472192d5416f45c5a1b1eee94531c363. Latest consensus state: current_term: 1 leader_uuid: "472192d5416f45c5a1b1eee94531c363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472192d5416f45c5a1b1eee94531c363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } } }
I20260927 10:53:57.134018 26930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:57.134454 26939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:57.135674 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472192d5416f45c5a1b1eee94531c363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472192d5416f45c5a1b1eee94531c363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } } }
I20260927 10:53:57.135854 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:57.136910 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:57.138276 26939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:57.146906 26939 catalog_manager.cc:1347] Generated new cluster ID: 4c546367813946c6bcb8c900eff93794
I20260927 10:53:57.147020 26939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:57.148821 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:57.139171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57097 (local address 127.0.0.1:44513)
0927 10:53:57.139310 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:57.139314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:57.139728 (+ 414us) server_negotiation.cc:407] Connection header received
0927 10:53:57.139853 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:57.139858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:57.139912 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:57.140006 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:57.140721 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.141562 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.142941 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.143276 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.143393 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:57.145193 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:57.145196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:57.145198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:57.145219 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:57.146951 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:57.147467 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:57.147471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:57.147474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:57.147579 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:57.147896 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:57.147899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:57.147901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:57.148136 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:57.148277 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:57.148520 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 10:53:57.148648 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:53:57.159744 26939 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:57.160629 26939 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:57.166329 26939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472192d5416f45c5a1b1eee94531c363: Generated new TSK 0
I20260927 10:53:57.167062 26939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:57.174080 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 26857
2026-09-27T10:53:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:57 dist-test-slave-c60x krb5kdc[26956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:57 dist-test-slave-c60x krb5kdc[26956](info): set up 2 sockets
Sep 27 10:53:57 dist-test-slave-c60x krb5kdc[26956](info): commencing operation
krb5kdc: starting...
W20260927 10:53:58.243499 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:53:58 dist-test-slave-c60x krb5kdc[26956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:53:58.367192 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34439
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:58.480475 26969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:58.480813 26969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:58.480911 26969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:58.485064 26969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:58.485157 26969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:58.485194 26969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:58.485256 26969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:58.485294 26969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:58.490658 26969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:58.491854 26969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:58.492923 26969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:58.501752 26974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:58.501767 26977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:58.501802 26975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:58.502074 26969 server_base.cc:1034] running on GCE node
I20260927 10:53:58.502612 26969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:58.503870 26969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:58.505131 26969 hybrid_clock.cc:648] HybridClock initialized: now 1790506438505108 us; error 31 us; skew 500 ppm
Sep 27 10:53:58 dist-test-slave-c60x krb5kdc[26956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:58.508080 26969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:53:58.508967 26969 webserver.cc:460] Webserver started at http://127.0.0.1:41569/ using document root <none> and password file <none>
I20260927 10:53:58.509537 26969 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:58.509629 26969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:58.509861 26969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:58.512022 26969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "5a3b87b7f722472e99ec8f41f1d83b19"
format_stamp: "Formatted at 2026-09-27 10:53:58 on dist-test-slave-c60x"
server_key: "0e4bbd2617a493311643784b15872b09"
server_key_iv: "cbc93ff23bd69ffff5270b38ebda96d0"
server_key_version: "encryptionkey@0"
I20260927 10:53:58.512591 26969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "5a3b87b7f722472e99ec8f41f1d83b19"
format_stamp: "Formatted at 2026-09-27 10:53:58 on dist-test-slave-c60x"
server_key: "0e4bbd2617a493311643784b15872b09"
server_key_iv: "cbc93ff23bd69ffff5270b38ebda96d0"
server_key_version: "encryptionkey@0"
I20260927 10:53:58.517908 26969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 10:53:58.523185 26984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:58.524833 26969 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 10:53:58.524976 26969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "5a3b87b7f722472e99ec8f41f1d83b19"
format_stamp: "Formatted at 2026-09-27 10:53:58 on dist-test-slave-c60x"
server_key: "0e4bbd2617a493311643784b15872b09"
server_key_iv: "cbc93ff23bd69ffff5270b38ebda96d0"
server_key_version: "encryptionkey@0"
I20260927 10:53:58.525100 26969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:58.550431 26969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:58.553885 26969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:58.554111 26969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:58.561769 26969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34439
I20260927 10:53:58.561827 27036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34439 every 8 connection(s)
I20260927 10:53:58.562755 26969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:58.566499 27037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:58.569592 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 26969
I20260927 10:53:58.569770 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:58.570129 6270 external_mini_cluster.cc:1428] Setting key 2461970c3d8eb91b3c6952613fad0123
I20260927 10:53:58.575109 27037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19: Bootstrap starting.
I20260927 10:53:58.578217 27037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:58.579267 27037 log.cc:824] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:58.579313 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:58.571573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55843 (local address 127.0.0.1:34439)
0927 10:53:58.571981 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:58.572006 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:58.572042 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:53:58.572702 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:58.572720 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:58.573062 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:58.573405 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:58.573835 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.574921 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.575932 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.576410 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.576602 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:58.577480 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:58.577485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:58.577497 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:58.577878 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:58.577931 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:58.577955 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:58.578066 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:58.578178 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 10:53:58.578483 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":133,"threads_started":1}
I20260927 10:53:58.581784 27037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19: No bootstrap required, opened a new log
I20260927 10:53:58.582201 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:58.585824 27037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3b87b7f722472e99ec8f41f1d83b19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34439 } }
I20260927 10:53:58.586054 27037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:58.586126 27037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3b87b7f722472e99ec8f41f1d83b19, State: Initialized, Role: FOLLOWER
I20260927 10:53:58.586580 27037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3b87b7f722472e99ec8f41f1d83b19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34439 } }
I20260927 10:53:58.586702 27037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:58.586790 27037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:58.586912 27037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:58.587929 27037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3b87b7f722472e99ec8f41f1d83b19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34439 } }
I20260927 10:53:58.588339 27037 leader_election.cc:302] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a3b87b7f722472e99ec8f41f1d83b19; no voters:
I20260927 10:53:58.588639 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:58.584325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50763 (local address 127.0.0.1:34439)
0927 10:53:58.584462 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:58.584467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:58.584688 (+ 221us) server_negotiation.cc:407] Connection header received
0927 10:53:58.584838 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:58.584841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:58.584883 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:58.584977 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:58.585288 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.585811 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.586859 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.587242 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.587351 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:58.588208 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:58.588212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:58.588214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:58.588227 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:58.588262 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:58.588274 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:58.588355 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:58.588445 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:53:58.588500 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:58.588730 27037 leader_election.cc:288] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:58.588903 27047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:58.589179 27047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [term 1 LEADER]: Becoming Leader. State: Replica: 5a3b87b7f722472e99ec8f41f1d83b19, State: Running, Role: LEADER
I20260927 10:53:58.589561 27047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3b87b7f722472e99ec8f41f1d83b19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34439 } }
I20260927 10:53:58.590102 27037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:58.590648 27047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3b87b7f722472e99ec8f41f1d83b19. Latest consensus state: current_term: 1 leader_uuid: "5a3b87b7f722472e99ec8f41f1d83b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3b87b7f722472e99ec8f41f1d83b19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34439 } } }
I20260927 10:53:58.590850 27047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:58.591101 27049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a3b87b7f722472e99ec8f41f1d83b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3b87b7f722472e99ec8f41f1d83b19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34439 } } }
I20260927 10:53:58.591229 27049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:58.592561 27056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:58.595325 27056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:58.600172 27056 catalog_manager.cc:1347] Generated new cluster ID: 6154f078030b4b98af6a28dea4325918
I20260927 10:53:58.600237 27056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:58.617395 27056 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:58.618685 27056 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:58.628679 27056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a3b87b7f722472e99ec8f41f1d83b19: Generated new TSK 0
I20260927 10:53:58.629491 27056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:58.635908 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 26969
2026-09-27T10:53:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:58 dist-test-slave-c60x krb5kdc[27068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:58 dist-test-slave-c60x krb5kdc[27068](info): set up 2 sockets
Sep 27 10:53:58 dist-test-slave-c60x krb5kdc[27068](info): commencing operation
krb5kdc: starting...
W20260927 10:53:59.679363 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:53:59 dist-test-slave-c60x krb5kdc[27068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:53:59.806774 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:59.918422 27081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:59.918700 27081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:59.918742 27081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:59.922842 27081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:59.922915 27081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:59.922935 27081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:59.922952 27081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:59.922968 27081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:59.928509 27081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:53:59.929682 27081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:59.930692 27081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:59.939339 27087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:59.939361 27089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:59.939539 27081 server_base.cc:1034] running on GCE node
W20260927 10:53:59.939340 27086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:59.940299 27081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:59.941484 27081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:59.942685 27081 hybrid_clock.cc:648] HybridClock initialized: now 1790506439942666 us; error 30 us; skew 500 ppm
Sep 27 10:53:59 dist-test-slave-c60x krb5kdc[27068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:59.945510 27081 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:53:59.946379 27081 webserver.cc:460] Webserver started at http://127.0.0.1:45067/ using document root <none> and password file <none>
I20260927 10:53:59.946909 27081 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:59.946966 27081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:59.947199 27081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:59.949210 27081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "1a451f5e726649faafff3844d02c9e5f"
format_stamp: "Formatted at 2026-09-27 10:53:59 on dist-test-slave-c60x"
server_key: "2222b8d8c59e033c2c2eb517c04b6137"
server_key_iv: "50387561c14d17a8581340eefbf06557"
server_key_version: "encryptionkey@0"
I20260927 10:53:59.949734 27081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "1a451f5e726649faafff3844d02c9e5f"
format_stamp: "Formatted at 2026-09-27 10:53:59 on dist-test-slave-c60x"
server_key: "2222b8d8c59e033c2c2eb517c04b6137"
server_key_iv: "50387561c14d17a8581340eefbf06557"
server_key_version: "encryptionkey@0"
I20260927 10:53:59.953491 27081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:53:59.956073 27096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:59.957398 27081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:59.957571 27081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "1a451f5e726649faafff3844d02c9e5f"
format_stamp: "Formatted at 2026-09-27 10:53:59 on dist-test-slave-c60x"
server_key: "2222b8d8c59e033c2c2eb517c04b6137"
server_key_iv: "50387561c14d17a8581340eefbf06557"
server_key_version: "encryptionkey@0"
I20260927 10:53:59.957687 27081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:59.962322 27081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:59.965479 27081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:59.965677 27081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:59.973158 27081 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35441
I20260927 10:53:59.973218 27148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35441 every 8 connection(s)
I20260927 10:53:59.974145 27081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:59.977026 27149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:59.982496 27149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f: Bootstrap starting.
I20260927 10:53:59.983340 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 27081
I20260927 10:53:59.983474 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:53:59.983825 6270 external_mini_cluster.cc:1428] Setting key 080892f2efb4291606049f3dea614b1d
I20260927 10:53:59.985486 27149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:59.986207 27149 log.cc:824] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:59.988523 27149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f: No bootstrap required, opened a new log
I20260927 10:53:59.989094 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:59.984947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59591 (local address 127.0.0.1:35441)
0927 10:53:59.985412 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:59.985427 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:59.985463 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:53:59.986181 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:59.986197 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:59.986570 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:59.986944 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:59.987794 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:59.987798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:59.987808 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:59.987817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:59.987865 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:59.987883 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:59.987970 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:59.988114 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 10:53:59.988283 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":133,"threads_started":1}
I20260927 10:53:59.991868 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:59.992264 27149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a451f5e726649faafff3844d02c9e5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35441 } }
I20260927 10:53:59.992552 27149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:59.992607 27149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a451f5e726649faafff3844d02c9e5f, State: Initialized, Role: FOLLOWER
I20260927 10:53:59.993081 27149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a451f5e726649faafff3844d02c9e5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35441 } }
I20260927 10:53:59.993258 27149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:59.993335 27149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:59.993455 27149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:59.994637 27149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a451f5e726649faafff3844d02c9e5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35441 } }
I20260927 10:53:59.995047 27149 leader_election.cc:302] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a451f5e726649faafff3844d02c9e5f; no voters:
I20260927 10:53:59.995486 27149 leader_election.cc:288] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:59.995592 27159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:59.995863 27159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [term 1 LEADER]: Becoming Leader. State: Replica: 1a451f5e726649faafff3844d02c9e5f, State: Running, Role: LEADER
I20260927 10:53:59.996166 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:59.994296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52805 (local address 127.0.0.1:35441)
0927 10:53:59.994428 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:59.994433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:59.994633 (+ 200us) server_negotiation.cc:407] Connection header received
0927 10:53:59.994761 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:59.994774 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:59.994818 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:59.994882 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:59.995681 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:59.995686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:59.995689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:59.995701 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:59.995731 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:59.995745 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:59.995837 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:59.996005 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 10:53:59.996046 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:53:59.996243 27159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a451f5e726649faafff3844d02c9e5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35441 } }
I20260927 10:53:59.996840 27149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:59.998402 27160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a451f5e726649faafff3844d02c9e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a451f5e726649faafff3844d02c9e5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35441 } } }
I20260927 10:53:59.998569 27160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:59.998849 27160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a451f5e726649faafff3844d02c9e5f. Latest consensus state: current_term: 1 leader_uuid: "1a451f5e726649faafff3844d02c9e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a451f5e726649faafff3844d02c9e5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35441 } } }
I20260927 10:53:59.999032 27160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f [sys.catalog]: This master's current role is: LEADER
W20260927 10:54:00.001677 27174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:00.001763 27174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:00.001943 27173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:00.003031 27173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:00.009032 27173 catalog_manager.cc:1347] Generated new cluster ID: f83d0e2301894cec8d842a185941c74f
I20260927 10:54:00.009106 27173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:00.026968 27173 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:00.027364 27173 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:00.045742 27173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a451f5e726649faafff3844d02c9e5f: Generated new TSK 0
I20260927 10:54:00.046705 27173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:00.059719 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 27081
2026-09-27T10:54:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:00 dist-test-slave-c60x krb5kdc[27180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:00 dist-test-slave-c60x krb5kdc[27180](info): set up 2 sockets
Sep 27 10:54:00 dist-test-slave-c60x krb5kdc[27180](info): commencing operation
krb5kdc: starting...
W20260927 10:54:01.103439 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:01 dist-test-slave-c60x krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:54:01.230964 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:01.339844 27193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:01.340175 27193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:01.340246 27193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:01.344203 27193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:01.344295 27193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:01.344333 27193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:01.344372 27193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:01.344406 27193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:01.349617 27193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:01.350664 27193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:01.351686 27193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:01.360236 27198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:01.360226 27201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:01.360226 27199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:01.360657 27193 server_base.cc:1034] running on GCE node
I20260927 10:54:01.361207 27193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:01.362298 27193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:01.363484 27193 hybrid_clock.cc:648] HybridClock initialized: now 1790506441363476 us; error 32 us; skew 500 ppm
Sep 27 10:54:01 dist-test-slave-c60x krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:01.366148 27193 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:54:01.367039 27193 webserver.cc:460] Webserver started at http://127.0.0.1:40465/ using document root <none> and password file <none>
I20260927 10:54:01.367621 27193 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:01.367730 27193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:01.367957 27193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:01.369967 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "71a8983a850b41fe8d35d479f2503948"
format_stamp: "Formatted at 2026-09-27 10:54:01 on dist-test-slave-c60x"
server_key: "70256f7a78c9cac6207c8a1c183bb2eb"
server_key_iv: "36064b8c222fb4d3a1554e754f2b0cdf"
server_key_version: "encryptionkey@0"
I20260927 10:54:01.370764 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "71a8983a850b41fe8d35d479f2503948"
format_stamp: "Formatted at 2026-09-27 10:54:01 on dist-test-slave-c60x"
server_key: "70256f7a78c9cac6207c8a1c183bb2eb"
server_key_iv: "36064b8c222fb4d3a1554e754f2b0cdf"
server_key_version: "encryptionkey@0"
I20260927 10:54:01.374593 27193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:54:01.377130 27208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:01.378271 27193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:54:01.378415 27193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "71a8983a850b41fe8d35d479f2503948"
format_stamp: "Formatted at 2026-09-27 10:54:01 on dist-test-slave-c60x"
server_key: "70256f7a78c9cac6207c8a1c183bb2eb"
server_key_iv: "36064b8c222fb4d3a1554e754f2b0cdf"
server_key_version: "encryptionkey@0"
I20260927 10:54:01.378531 27193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:01.388706 27193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:01.391615 27193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:01.391816 27193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:01.399011 27193 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41383
I20260927 10:54:01.399034 27260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41383 every 8 connection(s)
I20260927 10:54:01.399927 27193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:01.403986 27261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:01.407364 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 27193
I20260927 10:54:01.407523 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:01.407784 6270 external_mini_cluster.cc:1428] Setting key 5a0f455052e3e0ec0a56a036321198c1
I20260927 10:54:01.410346 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948: Bootstrap starting.
I20260927 10:54:01.413028 27261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:01.413858 27261 log.cc:824] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:01.416266 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948: No bootstrap required, opened a new log
Sep 27 10:54:01 dist-test-slave-c60x krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:54:01.419843 27261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8983a850b41fe8d35d479f2503948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } }
I20260927 10:54:01.420109 27261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:01.420198 27261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a8983a850b41fe8d35d479f2503948, State: Initialized, Role: FOLLOWER
I20260927 10:54:01.420687 27261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8983a850b41fe8d35d479f2503948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } }
I20260927 10:54:01.420838 27261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:01.420924 27261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:01.421023 27261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:01.421939 27261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8983a850b41fe8d35d479f2503948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } }
I20260927 10:54:01.422297 27261 leader_election.cc:302] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71a8983a850b41fe8d35d479f2503948; no voters:
I20260927 10:54:01.422668 27261 leader_election.cc:288] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:01.422777 27266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:01.422992 27266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [term 1 LEADER]: Becoming Leader. State: Replica: 71a8983a850b41fe8d35d479f2503948, State: Running, Role: LEADER
I20260927 10:54:01.423374 27266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8983a850b41fe8d35d479f2503948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } }
I20260927 10:54:01.423825 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:01.409231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:53879 (local address 127.0.0.1:41383)
0927 10:54:01.409642 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:01.409660 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:01.409701 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:54:01.410512 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:01.410534 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:01.410944 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:01.411318 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:01.412005 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.413495 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.414395 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.414987 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.415188 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:01.417895 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:01.417900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:01.417913 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:01.417939 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:01.420460 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:01.421033 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:01.421041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:01.421052 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:01.421214 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:01.421582 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:01.421585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:01.421587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:01.422002 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:01.422261 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:01.422605 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 10:54:01.422875 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":103,"threads_started":1}
I20260927 10:54:01.424021 27261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:01.425077 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71a8983a850b41fe8d35d479f2503948. Latest consensus state: current_term: 1 leader_uuid: "71a8983a850b41fe8d35d479f2503948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8983a850b41fe8d35d479f2503948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } } }
I20260927 10:54:01.425217 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:01.425513 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71a8983a850b41fe8d35d479f2503948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8983a850b41fe8d35d479f2503948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } } }
I20260927 10:54:01.425596 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:01.425614 27275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:01.427838 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:01.428797 27275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:01.435142 27275 catalog_manager.cc:1347] Generated new cluster ID: bb9f826c73a04ead8c15549bd218c9e5
I20260927 10:54:01.435274 27275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:01.438622 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:01.430055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:54901 (local address 127.0.0.1:41383)
0927 10:54:01.430161 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:01.430167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:01.430438 (+ 271us) server_negotiation.cc:407] Connection header received
0927 10:54:01.430546 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:01.430550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:01.430601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:01.430689 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:01.431479 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.431999 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.432810 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.433074 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.433183 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:01.435219 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:01.435223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:01.435226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:01.435250 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:01.436840 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:01.437336 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:01.437340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:01.437342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:01.437427 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:01.437755 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:01.437758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:01.437760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:01.437932 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:01.438108 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:01.438351 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 10:54:01.438446 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 10:54:01.441998 27275 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:01.442943 27275 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:01.450369 27275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71a8983a850b41fe8d35d479f2503948: Generated new TSK 0
I20260927 10:54:01.451414 27275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:01.460752 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 27193
W20260927 10:54:01.466167 27178 connection.cc:535] client connection to 127.0.0.1:41383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41383): Connection reset by peer (error 104)
2026-09-27T10:54:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:01 dist-test-slave-c60x krb5kdc[27292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:01 dist-test-slave-c60x krb5kdc[27292](info): set up 2 sockets
Sep 27 10:54:01 dist-test-slave-c60x krb5kdc[27292](info): commencing operation
krb5kdc: starting...
W20260927 10:54:02.499455 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:02 dist-test-slave-c60x krb5kdc[27292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:54:02.622694 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:02.731791 27305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:02.732089 27305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:02.732177 27305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:02.736212 27305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:02.736302 27305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:02.736341 27305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:02.736398 27305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:02.736438 27305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:02.741670 27305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:02.742726 27305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:02.743786 27305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:02.752020 27310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:02.752017 27311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:02.752177 27313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:02.752295 27305 server_base.cc:1034] running on GCE node
I20260927 10:54:02.752926 27305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:02.754065 27305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:02.755246 27305 hybrid_clock.cc:648] HybridClock initialized: now 1790506442755215 us; error 42 us; skew 500 ppm
Sep 27 10:54:02 dist-test-slave-c60x krb5kdc[27292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:02.757975 27305 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:54:02.758846 27305 webserver.cc:460] Webserver started at http://127.0.0.1:46615/ using document root <none> and password file <none>
I20260927 10:54:02.759481 27305 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:02.759577 27305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:02.759824 27305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:02.761852 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "5419a8765b8f472899ad0f36004df88d"
format_stamp: "Formatted at 2026-09-27 10:54:02 on dist-test-slave-c60x"
server_key: "59a7b51908d5de650ab4df0418173f75"
server_key_iv: "d7e658ab552108b0bd78dc008e821008"
server_key_version: "encryptionkey@0"
I20260927 10:54:02.762429 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "5419a8765b8f472899ad0f36004df88d"
format_stamp: "Formatted at 2026-09-27 10:54:02 on dist-test-slave-c60x"
server_key: "59a7b51908d5de650ab4df0418173f75"
server_key_iv: "d7e658ab552108b0bd78dc008e821008"
server_key_version: "encryptionkey@0"
I20260927 10:54:02.766254 27305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:54:02.768749 27320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:02.769925 27305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:02.770068 27305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "5419a8765b8f472899ad0f36004df88d"
format_stamp: "Formatted at 2026-09-27 10:54:02 on dist-test-slave-c60x"
server_key: "59a7b51908d5de650ab4df0418173f75"
server_key_iv: "d7e658ab552108b0bd78dc008e821008"
server_key_version: "encryptionkey@0"
I20260927 10:54:02.770188 27305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:02.784257 27305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:02.787444 27305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:02.787662 27305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:02.794649 27305 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43843
I20260927 10:54:02.794674 27372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43843 every 8 connection(s)
I20260927 10:54:02.795662 27305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:02.798453 27373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:02.798938 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 27305
I20260927 10:54:02.799068 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:02.799377 6270 external_mini_cluster.cc:1428] Setting key 738d9f3322fff44f209ef52e323d155f
I20260927 10:54:02.805817 27373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d: Bootstrap starting.
I20260927 10:54:02.808595 27373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:02.809490 27373 log.cc:824] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:02.809566 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:02.800829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:59601 (local address 127.0.0.1:43843)
0927 10:54:02.801265 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:02.801283 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:02.801328 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:54:02.802101 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:02.802124 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:02.802567 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:02.802945 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:02.803385 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.804887 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.805752 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.806409 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.806562 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:02.807506 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:02.807512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:02.807527 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:02.808023 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:02.808094 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:02.808125 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:02.808266 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:02.808428 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 10:54:02.808571 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260927 10:54:02.811585 27373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d: No bootstrap required, opened a new log
I20260927 10:54:02.812433 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:02.815449 27373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5419a8765b8f472899ad0f36004df88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260927 10:54:02.815673 27373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:02.815732 27373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5419a8765b8f472899ad0f36004df88d, State: Initialized, Role: FOLLOWER
I20260927 10:54:02.816187 27373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5419a8765b8f472899ad0f36004df88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260927 10:54:02.816315 27373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:02.816401 27373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:02.816527 27373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:02.817488 27373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5419a8765b8f472899ad0f36004df88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260927 10:54:02.817838 27373 leader_election.cc:302] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5419a8765b8f472899ad0f36004df88d; no voters:
I20260927 10:54:02.818183 27373 leader_election.cc:288] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:02.818254 27383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:02.818528 27383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [term 1 LEADER]: Becoming Leader. State: Replica: 5419a8765b8f472899ad0f36004df88d, State: Running, Role: LEADER
I20260927 10:54:02.818778 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:02.814406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:48135 (local address 127.0.0.1:43843)
0927 10:54:02.814541 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:02.814547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:02.814743 (+ 196us) server_negotiation.cc:407] Connection header received
0927 10:54:02.814885 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:02.814889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:02.814929 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:02.815012 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:02.815306 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.815921 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.816950 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.817207 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.817312 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:02.818193 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:02.818196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:02.818199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:02.818220 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:02.818264 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:02.818279 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:02.818367 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:02.818589 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 10:54:02.818636 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:54:02.818819 27383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5419a8765b8f472899ad0f36004df88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260927 10:54:02.819621 27373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:02.820019 27383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5419a8765b8f472899ad0f36004df88d. Latest consensus state: current_term: 1 leader_uuid: "5419a8765b8f472899ad0f36004df88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5419a8765b8f472899ad0f36004df88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } } }
I20260927 10:54:02.820130 27383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:02.820317 27383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5419a8765b8f472899ad0f36004df88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5419a8765b8f472899ad0f36004df88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } } }
I20260927 10:54:02.820415 27383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:02.820605 27387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:02.823834 27387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:02.830866 27387 catalog_manager.cc:1347] Generated new cluster ID: 4857c5e61d114050911cbd5941c00659
I20260927 10:54:02.830948 27387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:02.844946 27387 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:02.845925 27387 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:02.855242 27387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5419a8765b8f472899ad0f36004df88d: Generated new TSK 0
I20260927 10:54:02.855974 27387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:02.885136 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 27305
W20260927 10:54:02.891575 27290 connection.cc:535] client connection to 127.0.0.1:43843 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43843): Connection reset by peer (error 104)
2026-09-27T10:54:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:02 dist-test-slave-c60x krb5kdc[27404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:02 dist-test-slave-c60x krb5kdc[27404](info): set up 2 sockets
Sep 27 10:54:02 dist-test-slave-c60x krb5kdc[27404](info): commencing operation
krb5kdc: starting...
W20260927 10:54:03.967414 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:04 dist-test-slave-c60x krb5kdc[27404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:54:04.094717 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:04.203029 27417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:04.203372 27417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:04.203461 27417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:04.207585 27417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:04.207669 27417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:04.207708 27417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:04.207764 27417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:04.207805 27417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:04.213069 27417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:04.214181 27417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:04.215191 27417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:04.223415 27425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:04.223431 27423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:04.223429 27422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:04.223690 27417 server_base.cc:1034] running on GCE node
I20260927 10:54:04.224362 27417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:04.225458 27417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:04.226641 27417 hybrid_clock.cc:648] HybridClock initialized: now 1790506444226616 us; error 42 us; skew 500 ppm
Sep 27 10:54:04 dist-test-slave-c60x krb5kdc[27404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:04.229373 27417 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:54:04.230216 27417 webserver.cc:460] Webserver started at http://127.0.0.1:41979/ using document root <none> and password file <none>
I20260927 10:54:04.230723 27417 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:04.230779 27417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:04.230933 27417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:04.232937 27417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f"
format_stamp: "Formatted at 2026-09-27 10:54:04 on dist-test-slave-c60x"
server_key: "eada483c13b50e62c2ac998a7aba6643"
server_key_iv: "2b21fc95da93087d133532ad7449235c"
server_key_version: "encryptionkey@0"
I20260927 10:54:04.233425 27417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f"
format_stamp: "Formatted at 2026-09-27 10:54:04 on dist-test-slave-c60x"
server_key: "eada483c13b50e62c2ac998a7aba6643"
server_key_iv: "2b21fc95da93087d133532ad7449235c"
server_key_version: "encryptionkey@0"
I20260927 10:54:04.236877 27417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:54:04.239241 27432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:04.240372 27417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:04.240468 27417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f"
format_stamp: "Formatted at 2026-09-27 10:54:04 on dist-test-slave-c60x"
server_key: "eada483c13b50e62c2ac998a7aba6643"
server_key_iv: "2b21fc95da93087d133532ad7449235c"
server_key_version: "encryptionkey@0"
I20260927 10:54:04.240546 27417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:04.244863 27417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:04.247869 27417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:04.248003 27417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:04.254513 27417 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46715
I20260927 10:54:04.254537 27484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46715 every 8 connection(s)
I20260927 10:54:04.255411 27417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:04.258085 27485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:04.260474 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 27417
I20260927 10:54:04.260609 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:04.260908 6270 external_mini_cluster.cc:1428] Setting key c0f06216399f2448e886b3a050904c69
I20260927 10:54:04.263974 27485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f: Bootstrap starting.
I20260927 10:54:04.265859 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:04.262266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:59343 (local address 127.0.0.1:46715)
0927 10:54:04.262688 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:04.262703 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:04.262736 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:54:04.263401 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:04.263418 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:04.263687 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:04.263998 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:04.264755 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:04.264760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:04.264771 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:04.264780 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:04.264828 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:04.264846 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:04.265004 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:04.265070 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 10:54:04.265220 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":137,"threads_started":1}
I20260927 10:54:04.266343 27485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:04.267204 27485 log.cc:824] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:04.268627 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:04.269320 27485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f: No bootstrap required, opened a new log
I20260927 10:54:04.272256 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:04.270679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:50107 (local address 127.0.0.1:46715)
0927 10:54:04.270788 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:04.270792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:04.270918 (+ 126us) server_negotiation.cc:407] Connection header received
0927 10:54:04.271029 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:04.271031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:04.271056 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:04.271173 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:04.271854 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:04.271857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:04.271860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:04.271870 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:04.271894 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:04.271913 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:04.271996 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:04.272102 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 10:54:04.272140 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:54:04.273394 27485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46715 } }
I20260927 10:54:04.273604 27485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:04.273679 27485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6a0caa8e4048be9b66512ab3c2fa4f, State: Initialized, Role: FOLLOWER
I20260927 10:54:04.274076 27485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46715 } }
I20260927 10:54:04.274194 27485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:04.274295 27485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:04.274407 27485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:04.275329 27485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46715 } }
I20260927 10:54:04.275646 27485 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e6a0caa8e4048be9b66512ab3c2fa4f; no voters:
I20260927 10:54:04.275926 27485 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:04.276033 27495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:04.276268 27495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [term 1 LEADER]: Becoming Leader. State: Replica: 6e6a0caa8e4048be9b66512ab3c2fa4f, State: Running, Role: LEADER
I20260927 10:54:04.276537 27495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46715 } }
I20260927 10:54:04.277253 27485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:04.278167 27496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46715 } } }
I20260927 10:54:04.278311 27496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:04.278564 27497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6a0caa8e4048be9b66512ab3c2fa4f. Latest consensus state: current_term: 1 leader_uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6a0caa8e4048be9b66512ab3c2fa4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46715 } } }
I20260927 10:54:04.278692 27497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:04.278718 27504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:04.281507 27504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:04.286284 27504 catalog_manager.cc:1347] Generated new cluster ID: e980c20d8c3e46b98fab5ece43167f0b
I20260927 10:54:04.286347 27504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:04.320015 27504 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:04.320379 27504 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:04.327324 27504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6a0caa8e4048be9b66512ab3c2fa4f: Generated new TSK 0
I20260927 10:54:04.328135 27504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:04.339418 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 27417
2026-09-27T10:54:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:04 dist-test-slave-c60x krb5kdc[27516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:04 dist-test-slave-c60x krb5kdc[27516](info): set up 2 sockets
Sep 27 10:54:04 dist-test-slave-c60x krb5kdc[27516](info): commencing operation
krb5kdc: starting...
W20260927 10:54:05.383594 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:05 dist-test-slave-c60x krb5kdc[27516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:54:05.528458 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:05.649356 27529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:05.649662 27529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:05.649714 27529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:05.653788 27529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:05.653860 27529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:05.653882 27529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:05.653898 27529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:05.653913 27529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:05.659942 27529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:05.661715 27529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:05.663156 27529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:05.673192 27537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:05.673245 27534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:05.673272 27535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:05.673462 27529 server_base.cc:1034] running on GCE node
I20260927 10:54:05.674234 27529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:05.675549 27529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:05.676788 27529 hybrid_clock.cc:648] HybridClock initialized: now 1790506445676743 us; error 58 us; skew 500 ppm
Sep 27 10:54:05 dist-test-slave-c60x krb5kdc[27516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:05.680126 27529 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:54:05.681237 27529 webserver.cc:460] Webserver started at http://127.0.0.1:43521/ using document root <none> and password file <none>
I20260927 10:54:05.681907 27529 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:05.682015 27529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:05.682260 27529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:05.685490 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "5318f731889d4d049c4d83ede7822c45"
format_stamp: "Formatted at 2026-09-27 10:54:05 on dist-test-slave-c60x"
server_key: "c7c46f786a92059c8d0699c9521c99e4"
server_key_iv: "fc65d7730c4f511cad4fc737ae915054"
server_key_version: "encryptionkey@0"
I20260927 10:54:05.686275 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "5318f731889d4d049c4d83ede7822c45"
format_stamp: "Formatted at 2026-09-27 10:54:05 on dist-test-slave-c60x"
server_key: "c7c46f786a92059c8d0699c9521c99e4"
server_key_iv: "fc65d7730c4f511cad4fc737ae915054"
server_key_version: "encryptionkey@0"
I20260927 10:54:05.690955 27529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 10:54:05.695032 27544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:05.697013 27529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 10:54:05.697218 27529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "5318f731889d4d049c4d83ede7822c45"
format_stamp: "Formatted at 2026-09-27 10:54:05 on dist-test-slave-c60x"
server_key: "c7c46f786a92059c8d0699c9521c99e4"
server_key_iv: "fc65d7730c4f511cad4fc737ae915054"
server_key_version: "encryptionkey@0"
I20260927 10:54:05.697371 27529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:05.720245 27529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:05.723858 27529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:05.724086 27529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:05.731906 27529 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43977
I20260927 10:54:05.732412 27596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43977 every 8 connection(s)
I20260927 10:54:05.733018 27529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:05.737162 27597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:05.737190 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 27529
I20260927 10:54:05.737303 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:05.737640 6270 external_mini_cluster.cc:1428] Setting key edee455240b82fb6a72cb3e37836b3ce
I20260927 10:54:05.746026 27597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45: Bootstrap starting.
Sep 27 10:54:05 dist-test-slave-c60x krb5kdc[27516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:54:05.749810 27597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:05.750875 27597 log.cc:824] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:05.753227 27597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45: No bootstrap required, opened a new log
I20260927 10:54:05.755288 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:05.739496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:49303 (local address 127.0.0.1:43977)
0927 10:54:05.739997 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:05.740016 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:05.740063 (+ 47us) server_negotiation.cc:407] Connection header received
0927 10:54:05.741051 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:05.741138 (+ 87us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:05.741649 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:05.742074 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:05.742884 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.744416 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.745430 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.746129 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.746294 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:05.749918 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:05.749924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:05.749942 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:05.749979 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:05.752192 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:05.752751 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:05.752756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:05.752761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:05.752871 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:05.753224 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:05.753230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:05.753236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:05.753685 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:05.753905 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:05.754221 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 10:54:05.754446 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":148,"threads_started":1}
I20260927 10:54:05.757522 27597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5318f731889d4d049c4d83ede7822c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260927 10:54:05.757809 27597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:05.757889 27597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5318f731889d4d049c4d83ede7822c45, State: Initialized, Role: FOLLOWER
I20260927 10:54:05.758379 27597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5318f731889d4d049c4d83ede7822c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260927 10:54:05.758577 27597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:05.758667 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:05.758678 27597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:05.758785 27597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:05.759951 27597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5318f731889d4d049c4d83ede7822c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260927 10:54:05.760334 27597 leader_election.cc:302] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5318f731889d4d049c4d83ede7822c45; no voters:
I20260927 10:54:05.760670 27597 leader_election.cc:288] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:05.760751 27606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:05.760937 27606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [term 1 LEADER]: Becoming Leader. State: Replica: 5318f731889d4d049c4d83ede7822c45, State: Running, Role: LEADER
I20260927 10:54:05.761231 27606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5318f731889d4d049c4d83ede7822c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260927 10:54:05.762036 27597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:05.762456 27608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5318f731889d4d049c4d83ede7822c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5318f731889d4d049c4d83ede7822c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } } }
I20260927 10:54:05.762485 27609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5318f731889d4d049c4d83ede7822c45. Latest consensus state: current_term: 1 leader_uuid: "5318f731889d4d049c4d83ede7822c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5318f731889d4d049c4d83ede7822c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } } }
I20260927 10:54:05.762634 27609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:05.762635 27608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:05.763248 27612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:05.766806 27612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:05.770980 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:05.760832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:51639 (local address 127.0.0.1:43977)
0927 10:54:05.760984 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:05.760991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:05.761265 (+ 274us) server_negotiation.cc:407] Connection header received
0927 10:54:05.761445 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:05.761451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:05.761505 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:05.761579 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:05.762555 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.763350 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.764529 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.764878 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.764985 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:05.766673 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:05.766677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:05.766679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:05.766707 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:05.768862 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:05.769455 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:05.769459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:05.769461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:05.769551 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:05.769915 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:05.769919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:05.769921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:05.770137 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:05.770283 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:05.770653 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 10:54:05.770763 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 10:54:05.773414 27612 catalog_manager.cc:1347] Generated new cluster ID: 3309019a9e6c48eb82d5da7a955e31a9
I20260927 10:54:05.773481 27612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:05.801023 27612 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:05.802390 27612 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:05.809783 27612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5318f731889d4d049c4d83ede7822c45: Generated new TSK 0
I20260927 10:54:05.810989 27612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:05.819810 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 27529
W20260927 10:54:05.826146 27514 connection.cc:535] client connection to 127.0.0.1:43977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43977): Connection reset by peer (error 104)
2026-09-27T10:54:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:05 dist-test-slave-c60x krb5kdc[27628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:05 dist-test-slave-c60x krb5kdc[27628](info): set up 2 sockets
Sep 27 10:54:05 dist-test-slave-c60x krb5kdc[27628](info): commencing operation
krb5kdc: starting...
W20260927 10:54:06.900422 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:06 dist-test-slave-c60x krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:54:07.051736 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:07.184854 27641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:07.185209 27641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:07.185290 27641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:07.190198 27641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:07.190318 27641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:07.190367 27641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:07.190407 27641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:07.190444 27641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:07.198271 27641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:07.199611 27641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:07.200815 27641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:54:07.210322 27641 server_base.cc:1034] running on GCE node
W20260927 10:54:07.210239 27649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:07.210664 27646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:07.210239 27647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:07.211606 27641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:07.213214 27641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:07.214610 27641 hybrid_clock.cc:648] HybridClock initialized: now 1790506447214532 us; error 98 us; skew 500 ppm
Sep 27 10:54:07 dist-test-slave-c60x krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:07.218616 27641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:54:07.219996 27641 webserver.cc:460] Webserver started at http://127.0.0.1:33717/ using document root <none> and password file <none>
I20260927 10:54:07.220728 27641 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:07.220845 27641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:07.221131 27641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:07.223887 27641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "708c5c50e0f742a2899203d0e1331527"
format_stamp: "Formatted at 2026-09-27 10:54:07 on dist-test-slave-c60x"
server_key: "c85d2dea814068746c116926a781895e"
server_key_iv: "4c3af9823d3cea7e1f770b2458679ddf"
server_key_version: "encryptionkey@0"
I20260927 10:54:07.224606 27641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "708c5c50e0f742a2899203d0e1331527"
format_stamp: "Formatted at 2026-09-27 10:54:07 on dist-test-slave-c60x"
server_key: "c85d2dea814068746c116926a781895e"
server_key_iv: "4c3af9823d3cea7e1f770b2458679ddf"
server_key_version: "encryptionkey@0"
I20260927 10:54:07.230119 27641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 10:54:07.233744 27656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:07.235843 27641 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 10:54:07.236099 27641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "708c5c50e0f742a2899203d0e1331527"
format_stamp: "Formatted at 2026-09-27 10:54:07 on dist-test-slave-c60x"
server_key: "c85d2dea814068746c116926a781895e"
server_key_iv: "4c3af9823d3cea7e1f770b2458679ddf"
server_key_version: "encryptionkey@0"
I20260927 10:54:07.236406 27641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:07.256201 27641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:07.259843 27641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:07.260078 27641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:07.267987 27641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43259
I20260927 10:54:07.268008 27708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43259 every 8 connection(s)
I20260927 10:54:07.269230 27641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:07.271795 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 27641
I20260927 10:54:07.271947 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:07.272275 6270 external_mini_cluster.cc:1428] Setting key e27707c0ab6a425e463b430c8daba374
I20260927 10:54:07.273295 27709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:07.281525 27709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527: Bootstrap starting.
I20260927 10:54:07.284037 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:07.274247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:37295 (local address 127.0.0.1:43259)
0927 10:54:07.274803 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:07.274830 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:07.274894 (+ 64us) server_negotiation.cc:407] Connection header received
0927 10:54:07.275756 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:07.275778 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:07.276243 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:07.276645 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:07.277184 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.278466 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.279770 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.280266 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.280397 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:07.281442 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:07.281447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:07.281462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:07.281974 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:07.282036 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:07.282110 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:07.282226 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:07.282660 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 10:54:07.282899 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":188,"threads_started":1}
I20260927 10:54:07.284173 27709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:07.285184 27709 log.cc:824] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:07.287647 27709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527: No bootstrap required, opened a new log
I20260927 10:54:07.288378 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:07.292601 27709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708c5c50e0f742a2899203d0e1331527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260927 10:54:07.292907 27709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:07.292954 27709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708c5c50e0f742a2899203d0e1331527, State: Initialized, Role: FOLLOWER
I20260927 10:54:07.293440 27709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708c5c50e0f742a2899203d0e1331527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260927 10:54:07.293573 27709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:07.293617 27709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:07.293751 27709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:07.294881 27709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708c5c50e0f742a2899203d0e1331527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260927 10:54:07.295300 27709 leader_election.cc:302] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 708c5c50e0f742a2899203d0e1331527; no voters:
I20260927 10:54:07.295634 27709 leader_election.cc:288] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:07.295946 27719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:07.296183 27719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [term 1 LEADER]: Becoming Leader. State: Replica: 708c5c50e0f742a2899203d0e1331527, State: Running, Role: LEADER
I20260927 10:54:07.296489 27719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708c5c50e0f742a2899203d0e1331527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260927 10:54:07.297046 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:07.291260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:46191 (local address 127.0.0.1:43259)
0927 10:54:07.291425 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:07.291429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:07.291751 (+ 322us) server_negotiation.cc:407] Connection header received
0927 10:54:07.291978 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:07.291982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:07.292047 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:07.292148 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:07.292488 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.293237 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.294465 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.294855 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.294984 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:07.296436 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:07.296440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:07.296442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:07.296457 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:07.296486 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:07.296511 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:07.296572 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:07.296815 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 10:54:07.296897 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:54:07.297222 27709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:07.297791 27719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 708c5c50e0f742a2899203d0e1331527. Latest consensus state: current_term: 1 leader_uuid: "708c5c50e0f742a2899203d0e1331527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708c5c50e0f742a2899203d0e1331527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } } }
I20260927 10:54:07.297930 27719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:07.298379 27724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:07.298529 27720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "708c5c50e0f742a2899203d0e1331527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708c5c50e0f742a2899203d0e1331527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } } }
I20260927 10:54:07.298669 27720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:07.301542 27724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:07.308343 27724 catalog_manager.cc:1347] Generated new cluster ID: 8a31b32f6a15483f85bb6f65c210a840
I20260927 10:54:07.308446 27724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:07.320642 27724 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:07.321683 27724 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:07.326776 27724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 708c5c50e0f742a2899203d0e1331527: Generated new TSK 0
I20260927 10:54:07.327625 27724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:07.338807 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 27641
W20260927 10:54:07.344822 27626 connection.cc:535] client connection to 127.0.0.1:43259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43259): Connection reset by peer (error 104)
2026-09-27T10:54:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1518 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:07 dist-test-slave-c60x krb5kdc[27740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:07 dist-test-slave-c60x krb5kdc[27740](info): set up 2 sockets
Sep 27 10:54:07 dist-test-slave-c60x krb5kdc[27740](info): commencing operation
krb5kdc: starting...
W20260927 10:54:08.432684 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:08 dist-test-slave-c60x krb5kdc[27740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:54:08.572319 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:08.692474 27753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:08.692854 27753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:08.692934 27753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:08.697331 27753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:08.697455 27753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:08.697497 27753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:08.697567 27753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:08.697607 27753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:08.703708 27753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:08.704943 27753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:08.706038 27753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:08.715087 27759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:08.715183 27761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:08.715197 27758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:08.715528 27753 server_base.cc:1034] running on GCE node
I20260927 10:54:08.716105 27753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:08.717294 27753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:08.718487 27753 hybrid_clock.cc:648] HybridClock initialized: now 1790506448718431 us; error 56 us; skew 500 ppm
Sep 27 10:54:08 dist-test-slave-c60x krb5kdc[27740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:08.722244 27753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:54:08.723737 27753 webserver.cc:460] Webserver started at http://127.0.0.1:33855/ using document root <none> and password file <none>
I20260927 10:54:08.724583 27753 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:08.724677 27753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:08.724918 27753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:08.727953 27753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "ef133798ab044890a2272ca0fe1c3028"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-c60x"
server_key: "fc8a3168030ee6629d624a6f9f23abee"
server_key_iv: "10252079ffb7dc7df15022dc0ead7bbd"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.728679 27753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "ef133798ab044890a2272ca0fe1c3028"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-c60x"
server_key: "fc8a3168030ee6629d624a6f9f23abee"
server_key_iv: "10252079ffb7dc7df15022dc0ead7bbd"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.734087 27753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 10:54:08.737079 27768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.738899 27753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:08.739215 27753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "ef133798ab044890a2272ca0fe1c3028"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-c60x"
server_key: "fc8a3168030ee6629d624a6f9f23abee"
server_key_iv: "10252079ffb7dc7df15022dc0ead7bbd"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.739370 27753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:08.744338 27753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:08.748230 27753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:08.748481 27753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:08.756258 27753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41155
I20260927 10:54:08.756305 27820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41155 every 8 connection(s)
I20260927 10:54:08.757314 27753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:08.760257 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 27753
I20260927 10:54:08.760299 27821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:08.760413 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:08.760711 6270 external_mini_cluster.cc:1428] Setting key d6a01b422924cc48b7486045b50981c4
I20260927 10:54:08.766369 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:08.762152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:43467 (local address 127.0.0.1:41155)
0927 10:54:08.762637 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:08.762653 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:08.762693 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:54:08.763710 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:08.763730 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:08.764019 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:08.764363 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:08.765172 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:08.765177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:08.765188 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:08.765199 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:08.765250 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:08.765269 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:08.765423 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:08.765489 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 10:54:08.765657 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":140,"threads_started":1}
I20260927 10:54:08.767277 27821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028: Bootstrap starting.
I20260927 10:54:08.769439 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:08.769955 27821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:08.771001 27821 log.cc:824] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:08.773980 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:08.771767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:40663 (local address 127.0.0.1:41155)
0927 10:54:08.771905 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:08.771908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:08.772142 (+ 234us) server_negotiation.cc:407] Connection header received
0927 10:54:08.772481 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:08.772484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:08.772562 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:08.772679 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:08.773508 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:08.773512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:08.773514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:08.773526 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:08.773554 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:08.773567 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:08.773675 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:08.773804 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 10:54:08.773847 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:54:08.774236 27821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028: No bootstrap required, opened a new log
I20260927 10:54:08.778086 27821 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef133798ab044890a2272ca0fe1c3028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260927 10:54:08.778373 27821 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:08.778440 27821 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef133798ab044890a2272ca0fe1c3028, State: Initialized, Role: FOLLOWER
I20260927 10:54:08.778904 27821 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef133798ab044890a2272ca0fe1c3028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260927 10:54:08.779066 27821 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:08.779173 27821 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:08.779314 27821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:08.780472 27821 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef133798ab044890a2272ca0fe1c3028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260927 10:54:08.780839 27821 leader_election.cc:302] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef133798ab044890a2272ca0fe1c3028; no voters:
I20260927 10:54:08.781183 27821 leader_election.cc:288] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:08.781330 27831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:08.781575 27831 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [term 1 LEADER]: Becoming Leader. State: Replica: ef133798ab044890a2272ca0fe1c3028, State: Running, Role: LEADER
I20260927 10:54:08.781888 27831 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef133798ab044890a2272ca0fe1c3028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260927 10:54:08.782510 27821 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:08.783631 27832 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef133798ab044890a2272ca0fe1c3028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef133798ab044890a2272ca0fe1c3028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } } }
I20260927 10:54:08.783777 27832 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:08.784068 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef133798ab044890a2272ca0fe1c3028. Latest consensus state: current_term: 1 leader_uuid: "ef133798ab044890a2272ca0fe1c3028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef133798ab044890a2272ca0fe1c3028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } } }
I20260927 10:54:08.784148 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:08.784147 27840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:08.787422 27840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:08.793188 27840 catalog_manager.cc:1347] Generated new cluster ID: 0950a606f33942e7962660e210397a52
I20260927 10:54:08.793277 27840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:08.810528 27840 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:08.811120 27840 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:08.824388 27840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef133798ab044890a2272ca0fe1c3028: Generated new TSK 0
I20260927 10:54:08.825210 27840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:08.830700 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 27753
2026-09-27T10:54:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:08 dist-test-slave-c60x krb5kdc[27852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:08 dist-test-slave-c60x krb5kdc[27852](info): set up 2 sockets
Sep 27 10:54:08 dist-test-slave-c60x krb5kdc[27852](info): commencing operation
krb5kdc: starting...
W20260927 10:54:09.907815 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:10 dist-test-slave-c60x krb5kdc[27852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:54:10.058794 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:40917
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:40917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:10.181656 27865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:10.181980 27865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:10.182025 27865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:10.187574 27865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:10.187666 27865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:10.187687 27865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:10.187705 27865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:10.187721 27865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:10.193673 27865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:40917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:40917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:10.194893 27865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:10.196002 27865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:10.205590 27873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:10.205729 27865 server_base.cc:1034] running on GCE node
W20260927 10:54:10.205577 27870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:10.205585 27871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:10.206560 27865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:10.207764 27865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:10.208949 27865 hybrid_clock.cc:648] HybridClock initialized: now 1790506450208932 us; error 31 us; skew 500 ppm
Sep 27 10:54:10 dist-test-slave-c60x krb5kdc[27852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506450, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:10.212286 27865 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:54:10.213284 27865 webserver.cc:460] Webserver started at http://127.6.31.190:40345/ using document root <none> and password file <none>
I20260927 10:54:10.213930 27865 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:10.214030 27865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:10.214263 27865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:10.216616 27865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "557aa37344cd44b6971471864f9dc111"
format_stamp: "Formatted at 2026-09-27 10:54:10 on dist-test-slave-c60x"
server_key: "4b108bce405f0f94c6d16d42d8ae0552"
server_key_iv: "1815ecc4d668b74299841b135762beb0"
server_key_version: "encryptionkey@0"
I20260927 10:54:10.217272 27865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "557aa37344cd44b6971471864f9dc111"
format_stamp: "Formatted at 2026-09-27 10:54:10 on dist-test-slave-c60x"
server_key: "4b108bce405f0f94c6d16d42d8ae0552"
server_key_iv: "1815ecc4d668b74299841b135762beb0"
server_key_version: "encryptionkey@0"
I20260927 10:54:10.221387 27865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:54:10.224098 27880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:10.225464 27865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:10.225647 27865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "557aa37344cd44b6971471864f9dc111"
format_stamp: "Formatted at 2026-09-27 10:54:10 on dist-test-slave-c60x"
server_key: "4b108bce405f0f94c6d16d42d8ae0552"
server_key_iv: "1815ecc4d668b74299841b135762beb0"
server_key_version: "encryptionkey@0"
I20260927 10:54:10.225781 27865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:10.254290 27865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:10.257911 27865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:10.258129 27865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:10.266158 27865 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:40917
I20260927 10:54:10.266376 27932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:40917 every 8 connection(s)
I20260927 10:54:10.267434 27865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:10.270668 27933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:10.277173 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 27865
I20260927 10:54:10.277364 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:10.277678 6270 external_mini_cluster.cc:1428] Setting key 613aa1e46a7525beecfb4768f2842f78
I20260927 10:54:10.279414 27933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111: Bootstrap starting.
I20260927 10:54:10.282467 27933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:10.283509 27933 log.cc:824] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:10.286968 27933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111: No bootstrap required, opened a new log
Sep 27 10:54:10 dist-test-slave-c60x krb5kdc[27852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:54:10.292369 27933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557aa37344cd44b6971471864f9dc111" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40917 } }
I20260927 10:54:10.292680 27933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:10.292779 27933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557aa37344cd44b6971471864f9dc111, State: Initialized, Role: FOLLOWER
I20260927 10:54:10.293318 27933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557aa37344cd44b6971471864f9dc111" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40917 } }
I20260927 10:54:10.293517 27933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:10.293623 27933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:10.293732 27933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:10.295123 27933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557aa37344cd44b6971471864f9dc111" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40917 } }
I20260927 10:54:10.295575 27933 leader_election.cc:302] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 557aa37344cd44b6971471864f9dc111; no voters:
I20260927 10:54:10.295982 27933 leader_election.cc:288] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:10.296351 27938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:10.296505 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:10.279290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34221 (local address 127.6.31.190:40917)
0927 10:54:10.279776 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:10.279797 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:10.279842 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:54:10.280849 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:10.280874 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:10.281349 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:10.281760 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:10.282974 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:10.284464 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:10.285612 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:10.286191 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:10.286400 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:10.289509 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:10.289515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:10.289533 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:10.289570 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:10.292094 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:10.293110 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:10.293119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:10.293126 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:10.293302 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:10.293841 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:10.293846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:10.293848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:10.294433 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:10.294675 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:10.295180 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 10:54:10.295437 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":156,"threads_started":1}
I20260927 10:54:10.296737 27938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [term 1 LEADER]: Becoming Leader. State: Replica: 557aa37344cd44b6971471864f9dc111, State: Running, Role: LEADER
I20260927 10:54:10.297245 27938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557aa37344cd44b6971471864f9dc111" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40917 } }
I20260927 10:54:10.297417 27933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:10.299479 27939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "557aa37344cd44b6971471864f9dc111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557aa37344cd44b6971471864f9dc111" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40917 } } }
I20260927 10:54:10.299988 27939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:10.299754 27940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 557aa37344cd44b6971471864f9dc111. Latest consensus state: current_term: 1 leader_uuid: "557aa37344cd44b6971471864f9dc111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557aa37344cd44b6971471864f9dc111" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 40917 } } }
I20260927 10:54:10.300320 27940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:10.300673 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 10:54:10.303473 27952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:10.303572 27952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:10.303665 27953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:10.304840 27953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:10.313664 27953 catalog_manager.cc:1347] Generated new cluster ID: dceb112e015f45bda71818d3694ec7f1
I20260927 10:54:10.313766 27953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:10.315541 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:10.303133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57727 (local address 127.6.31.190:40917)
0927 10:54:10.303262 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:10.303267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:10.303770 (+ 503us) server_negotiation.cc:407] Connection header received
0927 10:54:10.305149 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:10.305153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:10.305220 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:10.305285 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:10.306207 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:10.306857 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:10.308086 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:10.308552 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:10.308733 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:10.310777 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:10.310781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:10.310783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:10.310817 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:10.313024 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:10.313818 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:10.313822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:10.313825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:10.313940 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:10.314352 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:10.314356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:10.314358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:10.314601 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:10.314798 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:10.315169 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 10:54:10.315324 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:54:10.332070 27953 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:10.333333 27953 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:10.341804 27953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 557aa37344cd44b6971471864f9dc111: Generated new TSK 0
I20260927 10:54:10.342986 27953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:10.354140 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 27865
W20260927 10:54:10.359831 27850 connection.cc:535] client connection to 127.6.31.190:40917 recv error: Network error: failed to read from TLS socket (remote: 127.6.31.190:40917): Connection reset by peer (error 104)
2026-09-27T10:54:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1523 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:10 dist-test-slave-c60x krb5kdc[27964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:10 dist-test-slave-c60x krb5kdc[27964](info): set up 2 sockets
Sep 27 10:54:10 dist-test-slave-c60x krb5kdc[27964](info): commencing operation
krb5kdc: starting...
W20260927 10:54:11.428160 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:11 dist-test-slave-c60x krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:54:11.563760 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:39629
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:39629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:11.678428 27977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:11.678788 27977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:11.678876 27977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:11.683151 27977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:11.683264 27977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:11.683305 27977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:11.683342 27977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:11.683379 27977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:11.689093 27977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:39629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:39629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:11.690316 27977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:11.691453 27977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:11.701630 27983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:11.701731 27985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:11.701853 27977 server_base.cc:1034] running on GCE node
W20260927 10:54:11.701622 27982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:11.702620 27977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:11.703924 27977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:11.705186 27977 hybrid_clock.cc:648] HybridClock initialized: now 1790506451705156 us; error 40 us; skew 500 ppm
Sep 27 10:54:11 dist-test-slave-c60x krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506451, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:11.709561 27977 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:54:11.710944 27977 webserver.cc:460] Webserver started at http://127.6.31.190:39275/ using document root <none> and password file <none>
I20260927 10:54:11.711872 27977 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:11.712162 27977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:11.712503 27977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:11.715634 27977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "cbf42f5f537b42caaaa82bd22bdcd6be"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-c60x"
server_key: "26a74d199a50df0596f4d9f801c3004a"
server_key_iv: "31eb833a97e6012f2bd361dd86bdfeb0"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.716296 27977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "cbf42f5f537b42caaaa82bd22bdcd6be"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-c60x"
server_key: "26a74d199a50df0596f4d9f801c3004a"
server_key_iv: "31eb833a97e6012f2bd361dd86bdfeb0"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.721220 27977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 10:54:11.725229 27992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.726871 27977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:11.727180 27977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "cbf42f5f537b42caaaa82bd22bdcd6be"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-c60x"
server_key: "26a74d199a50df0596f4d9f801c3004a"
server_key_iv: "31eb833a97e6012f2bd361dd86bdfeb0"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.727361 27977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:11.751317 27977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:11.756078 27977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:11.756345 27977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:11.764647 27977 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:39629
I20260927 10:54:11.764673 28044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:39629 every 8 connection(s)
I20260927 10:54:11.765974 27977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:11.769644 28045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:11.771950 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 27977
I20260927 10:54:11.772097 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:11.772470 6270 external_mini_cluster.cc:1428] Setting key 0c8d6733b07af52fbcdef3d22be92a60
I20260927 10:54:11.777951 28045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be: Bootstrap starting.
I20260927 10:54:11.780910 28045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:11.781842 28045 log.cc:824] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:11.784602 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.774331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49289 (local address 127.6.31.190:39629)
0927 10:54:11.774836 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.774858 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.774902 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:54:11.775897 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.775924 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.776390 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.776795 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:11.777244 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.778700 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.779729 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.780414 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.780583 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.781705 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:11.781715 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:11.781734 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:11.782312 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:11.782410 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:11.782437 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:11.782561 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.783040 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.783325 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":119,"threads_started":1}
I20260927 10:54:11.785272 28045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be: No bootstrap required, opened a new log
I20260927 10:54:11.788947 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:11.789805 28045 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf42f5f537b42caaaa82bd22bdcd6be" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39629 } }
I20260927 10:54:11.790058 28045 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:11.790122 28045 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf42f5f537b42caaaa82bd22bdcd6be, State: Initialized, Role: FOLLOWER
I20260927 10:54:11.790673 28045 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf42f5f537b42caaaa82bd22bdcd6be" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39629 } }
I20260927 10:54:11.790841 28045 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:11.790930 28045 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:11.791055 28045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:11.792415 28045 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf42f5f537b42caaaa82bd22bdcd6be" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39629 } }
I20260927 10:54:11.792801 28045 leader_election.cc:302] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf42f5f537b42caaaa82bd22bdcd6be; no voters:
I20260927 10:54:11.793154 28045 leader_election.cc:288] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:11.793495 28055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:11.793825 28055 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [term 1 LEADER]: Becoming Leader. State: Replica: cbf42f5f537b42caaaa82bd22bdcd6be, State: Running, Role: LEADER
I20260927 10:54:11.794180 28055 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf42f5f537b42caaaa82bd22bdcd6be" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39629 } }
I20260927 10:54:11.794595 28045 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:11.796152 28056 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf42f5f537b42caaaa82bd22bdcd6be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf42f5f537b42caaaa82bd22bdcd6be" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39629 } } }
I20260927 10:54:11.796285 28056 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:11.796631 28057 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf42f5f537b42caaaa82bd22bdcd6be. Latest consensus state: current_term: 1 leader_uuid: "cbf42f5f537b42caaaa82bd22bdcd6be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf42f5f537b42caaaa82bd22bdcd6be" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 39629 } } }
I20260927 10:54:11.796728 28057 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:11.797281 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.791230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45099 (local address 127.6.31.190:39629)
0927 10:54:11.791381 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.791386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.791696 (+ 310us) server_negotiation.cc:407] Connection header received
0927 10:54:11.791772 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.791775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.791848 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.791950 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:11.792432 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.793274 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.794746 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.795219 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.795326 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.796529 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:11.796533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:11.796538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:11.796556 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:11.796586 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:11.796601 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:11.796686 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.797088 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.797130 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:54:11.797390 28066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:11.800895 28066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:11.806924 28066 catalog_manager.cc:1347] Generated new cluster ID: 08316922ac3e4f7ca15fd7cb69f24d24
I20260927 10:54:11.807011 28066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:11.832100 28066 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:11.833211 28066 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:11.841661 28066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf42f5f537b42caaaa82bd22bdcd6be: Generated new TSK 0
I20260927 10:54:11.842494 28066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:11.848223 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 27977
W20260927 10:54:11.854589 27962 connection.cc:535] client connection to 127.6.31.190:39629 recv error: Network error: failed to read from TLS socket (remote: 127.6.31.190:39629): Connection reset by peer (error 104)
2026-09-27T10:54:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1494 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:11 dist-test-slave-c60x krb5kdc[28076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:11 dist-test-slave-c60x krb5kdc[28076](info): set up 2 sockets
Sep 27 10:54:11 dist-test-slave-c60x krb5kdc[28076](info): commencing operation
krb5kdc: starting...
W20260927 10:54:12.920049 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:13 dist-test-slave-c60x krb5kdc[28076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:54:13.059700 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35087
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:35087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:13.180347 28089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:13.180657 28089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:13.180706 28089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:13.187223 28089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:13.187340 28089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:13.187362 28089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:13.187382 28089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:13.187400 28089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:13.196770 28089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:35087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:35087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:13.198117 28089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:13.199342 28089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:13.211201 28097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:13.211201 28095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:13.211246 28094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:13.211772 28089 server_base.cc:1034] running on GCE node
I20260927 10:54:13.212457 28089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:13.213762 28089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:13.214993 28089 hybrid_clock.cc:648] HybridClock initialized: now 1790506453214941 us; error 47 us; skew 500 ppm
Sep 27 10:54:13 dist-test-slave-c60x krb5kdc[28076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506453, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:13.225571 28089 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:54:13.227306 28089 webserver.cc:460] Webserver started at http://127.6.31.190:42305/ using document root <none> and password file <none>
I20260927 10:54:13.228132 28089 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:13.228245 28089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:13.228556 28089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:13.231612 28089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "9e2dab2990884384bb70e3da04326b63"
format_stamp: "Formatted at 2026-09-27 10:54:13 on dist-test-slave-c60x"
server_key: "ed6cd6fbf0f45d4b61bd9bc3680279c5"
server_key_iv: "0841b36e32a425902e482f998bb686e9"
server_key_version: "encryptionkey@0"
I20260927 10:54:13.232782 28089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "9e2dab2990884384bb70e3da04326b63"
format_stamp: "Formatted at 2026-09-27 10:54:13 on dist-test-slave-c60x"
server_key: "ed6cd6fbf0f45d4b61bd9bc3680279c5"
server_key_iv: "0841b36e32a425902e482f998bb686e9"
server_key_version: "encryptionkey@0"
I20260927 10:54:13.238600 28089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 10:54:13.242750 28104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:13.245121 28089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:54:13.245558 28089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "9e2dab2990884384bb70e3da04326b63"
format_stamp: "Formatted at 2026-09-27 10:54:13 on dist-test-slave-c60x"
server_key: "ed6cd6fbf0f45d4b61bd9bc3680279c5"
server_key_iv: "0841b36e32a425902e482f998bb686e9"
server_key_version: "encryptionkey@0"
I20260927 10:54:13.245827 28089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:13.252982 28089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:13.256842 28089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:13.257090 28089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:13.265909 28089 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:35087
I20260927 10:54:13.265937 28156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:35087 every 8 connection(s)
I20260927 10:54:13.267645 28089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:13.271275 28157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:13.277632 28157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63: Bootstrap starting.
I20260927 10:54:13.277799 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 28089
I20260927 10:54:13.277953 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:13.278277 6270 external_mini_cluster.cc:1428] Setting key c746fcd1dade77614b97b1e9422853ef
I20260927 10:54:13.280766 28157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:13.281664 28157 log.cc:824] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:13.284040 28157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63: No bootstrap required, opened a new log
I20260927 10:54:13.285668 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:13.280092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56083 (local address 127.6.31.190:35087)
0927 10:54:13.280786 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:13.280818 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:13.281002 (+ 184us) server_negotiation.cc:407] Connection header received
0927 10:54:13.282157 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:13.282182 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:13.282577 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:13.283030 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:13.283984 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:13.283989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:13.284002 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:13.284012 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:13.284080 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:13.284129 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:13.284261 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:13.284470 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 10:54:13.284673 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":182,"threads_started":1}
I20260927 10:54:13.288698 28157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2dab2990884384bb70e3da04326b63" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35087 } }
I20260927 10:54:13.288995 28157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:13.289083 28157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2dab2990884384bb70e3da04326b63, State: Initialized, Role: FOLLOWER
I20260927 10:54:13.289625 28157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2dab2990884384bb70e3da04326b63" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35087 } }
I20260927 10:54:13.289834 28157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:13.289960 28157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:13.289952 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:13.290096 28157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:13.291226 28157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2dab2990884384bb70e3da04326b63" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35087 } }
I20260927 10:54:13.291627 28157 leader_election.cc:302] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e2dab2990884384bb70e3da04326b63; no voters:
I20260927 10:54:13.292084 28157 leader_election.cc:288] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:13.292148 28166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:13.292364 28166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2dab2990884384bb70e3da04326b63, State: Running, Role: LEADER
I20260927 10:54:13.293363 28166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2dab2990884384bb70e3da04326b63" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35087 } }
I20260927 10:54:13.293586 28157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:13.295195 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:13.292833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54843 (local address 127.6.31.190:35087)
0927 10:54:13.292972 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:13.292978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:13.293531 (+ 553us) server_negotiation.cc:407] Connection header received
0927 10:54:13.293590 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:13.293593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:13.293653 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:13.293755 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:13.294603 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:13.294606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:13.294609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:13.294631 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:13.294663 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:13.294678 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:13.294752 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:13.294980 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 10:54:13.295032 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:54:13.295300 28168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e2dab2990884384bb70e3da04326b63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2dab2990884384bb70e3da04326b63" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35087 } } }
I20260927 10:54:13.295456 28168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:13.296021 28167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e2dab2990884384bb70e3da04326b63. Latest consensus state: current_term: 1 leader_uuid: "9e2dab2990884384bb70e3da04326b63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2dab2990884384bb70e3da04326b63" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 35087 } } }
I20260927 10:54:13.296758 28167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63 [sys.catalog]: This master's current role is: LEADER
W20260927 10:54:13.298693 28182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:13.298791 28182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:13.298869 28181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:13.299765 28181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:13.307675 28181 catalog_manager.cc:1347] Generated new cluster ID: d86676474cfa46de9c93e9b81e04e215
I20260927 10:54:13.307778 28181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:13.329376 28181 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:13.329768 28181 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:13.334610 28181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e2dab2990884384bb70e3da04326b63: Generated new TSK 0
I20260927 10:54:13.335807 28181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:13.347064 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 28089
2026-09-27T10:54:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:13 dist-test-slave-c60x krb5kdc[28188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:13 dist-test-slave-c60x krb5kdc[28188](info): set up 2 sockets
Sep 27 10:54:13 dist-test-slave-c60x krb5kdc[28188](info): commencing operation
krb5kdc: starting...
W20260927 10:54:14.415596 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:14 dist-test-slave-c60x krb5kdc[28188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:54:14.550158 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46471
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:46471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:14.665227 28201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:14.665529 28201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:14.665573 28201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:14.669879 28201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:14.669961 28201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:14.669987 28201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:14.670006 28201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:14.670056 28201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:14.675822 28201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:46471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:46471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:14.677081 28201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:14.678213 28201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:14.687508 28207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:14.687520 28206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:14.687527 28201 server_base.cc:1034] running on GCE node
W20260927 10:54:14.687551 28209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:14.688361 28201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:14.689522 28201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:14.690706 28201 hybrid_clock.cc:648] HybridClock initialized: now 1790506454690691 us; error 30 us; skew 500 ppm
Sep 27 10:54:14 dist-test-slave-c60x krb5kdc[28188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506454, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:14.693696 28201 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:54:14.694698 28201 webserver.cc:460] Webserver started at http://127.6.31.190:38231/ using document root <none> and password file <none>
I20260927 10:54:14.695362 28201 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:14.695438 28201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:14.695622 28201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:14.697858 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "56a12608ef094e95987c5e1df35d366d"
format_stamp: "Formatted at 2026-09-27 10:54:14 on dist-test-slave-c60x"
server_key: "31038f38974aace3be08008527401d7a"
server_key_iv: "eea5b1b042f8124660f96c62ee2140cc"
server_key_version: "encryptionkey@0"
I20260927 10:54:14.698716 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "56a12608ef094e95987c5e1df35d366d"
format_stamp: "Formatted at 2026-09-27 10:54:14 on dist-test-slave-c60x"
server_key: "31038f38974aace3be08008527401d7a"
server_key_iv: "eea5b1b042f8124660f96c62ee2140cc"
server_key_version: "encryptionkey@0"
I20260927 10:54:14.702832 28201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:54:14.705579 28216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:14.707031 28201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 10:54:14.707234 28201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "56a12608ef094e95987c5e1df35d366d"
format_stamp: "Formatted at 2026-09-27 10:54:14 on dist-test-slave-c60x"
server_key: "31038f38974aace3be08008527401d7a"
server_key_iv: "eea5b1b042f8124660f96c62ee2140cc"
server_key_version: "encryptionkey@0"
I20260927 10:54:14.707366 28201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:14.720152 28201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:14.723492 28201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:14.723714 28201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:14.731206 28201 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:46471
I20260927 10:54:14.731233 28268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:46471 every 8 connection(s)
I20260927 10:54:14.732236 28201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:14.735157 28269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:14.737426 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 28201
I20260927 10:54:14.737556 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:14.737838 6270 external_mini_cluster.cc:1428] Setting key 1b29a512bd6086c994222aaf0d6a3750
I20260927 10:54:14.741492 28269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d: Bootstrap starting.
I20260927 10:54:14.744334 28269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:14.745177 28269 log.cc:824] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:54:14 dist-test-slave-c60x krb5kdc[28188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:54:14.747720 28269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d: No bootstrap required, opened a new log
I20260927 10:54:14.751724 28269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a12608ef094e95987c5e1df35d366d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46471 } }
I20260927 10:54:14.752000 28269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:14.752120 28269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a12608ef094e95987c5e1df35d366d, State: Initialized, Role: FOLLOWER
I20260927 10:54:14.752652 28269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a12608ef094e95987c5e1df35d366d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46471 } }
I20260927 10:54:14.752822 28269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:14.752882 28269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:14.752995 28269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:14.753633 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:14.739298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53199 (local address 127.6.31.190:46471)
0927 10:54:14.739800 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:14.739816 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:14.739850 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:54:14.740588 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:14.740606 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:14.740977 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:14.741268 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:14.742205 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:14.743286 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:14.744569 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:14.745085 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:14.745210 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:14.748363 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:14.748369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:14.748385 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:14.748413 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:14.750212 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:14.750723 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:14.750729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:14.750734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:14.750851 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:14.751188 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:14.751192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:14.751193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:14.751662 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:14.751885 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:14.752200 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 10:54:14.752515 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":183,"threads_started":1}
I20260927 10:54:14.754345 28269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a12608ef094e95987c5e1df35d366d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46471 } }
I20260927 10:54:14.754706 28269 leader_election.cc:302] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56a12608ef094e95987c5e1df35d366d; no voters:
I20260927 10:54:14.755055 28269 leader_election.cc:288] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:14.755218 28274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:14.755486 28274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [term 1 LEADER]: Becoming Leader. State: Replica: 56a12608ef094e95987c5e1df35d366d, State: Running, Role: LEADER
I20260927 10:54:14.755889 28274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a12608ef094e95987c5e1df35d366d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46471 } }
I20260927 10:54:14.756373 28269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:14.757453 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:14.757900 28275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56a12608ef094e95987c5e1df35d366d. Latest consensus state: current_term: 1 leader_uuid: "56a12608ef094e95987c5e1df35d366d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a12608ef094e95987c5e1df35d366d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46471 } } }
I20260927 10:54:14.757910 28276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56a12608ef094e95987c5e1df35d366d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a12608ef094e95987c5e1df35d366d" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 46471 } } }
I20260927 10:54:14.758033 28275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:14.758050 28276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:14.758436 28285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:14.761834 28285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:14.768978 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:14.759754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41247 (local address 127.6.31.190:46471)
0927 10:54:14.759860 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:14.759863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:14.760058 (+ 195us) server_negotiation.cc:407] Connection header received
0927 10:54:14.760240 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:14.760243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:14.760292 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:14.760391 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:14.761258 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:14.761808 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:14.762598 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:14.762844 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:14.762957 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:14.764611 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:14.764616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:14.764619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:14.764650 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:14.767000 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:14.767601 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:14.767605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:14.767606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:14.767713 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:14.768050 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:14.768054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:14.768055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:14.768295 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:14.768435 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:14.768671 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 10:54:14.768808 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 10:54:14.770648 28285 catalog_manager.cc:1347] Generated new cluster ID: 7b33f41e7afc40b3acb70e77c0ca7512
I20260927 10:54:14.770723 28285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:14.792481 28285 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:14.793820 28285 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:14.801172 28285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56a12608ef094e95987c5e1df35d366d: Generated new TSK 0
I20260927 10:54:14.802069 28285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:14.813457 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 28201
2026-09-27T10:54:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:14 dist-test-slave-c60x krb5kdc[28300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:14 dist-test-slave-c60x krb5kdc[28300](info): set up 2 sockets
Sep 27 10:54:14 dist-test-slave-c60x krb5kdc[28300](info): commencing operation
krb5kdc: starting...
W20260927 10:54:15.859442 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:15 dist-test-slave-c60x krb5kdc[28300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:54:15.982651 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36349
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:16.093922 28313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:16.094297 28313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:16.094383 28313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:16.098596 28313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:16.098701 28313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:16.098742 28313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:16.098799 28313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:16.098855 28313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:16.104409 28313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:16.105547 28313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:16.106603 28313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:16.114933 28318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:16.114933 28319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:16.114938 28321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:16.115302 28313 server_base.cc:1034] running on GCE node
I20260927 10:54:16.115861 28313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:16.116942 28313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:16.118165 28313 hybrid_clock.cc:648] HybridClock initialized: now 1790506456118144 us; error 37 us; skew 500 ppm
Sep 27 10:54:16 dist-test-slave-c60x krb5kdc[28300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:16.121307 28313 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:54:16.122305 28313 webserver.cc:460] Webserver started at http://127.6.31.190:42957/ using document root <none> and password file <none>
I20260927 10:54:16.122915 28313 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:16.122979 28313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:16.123253 28313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:16.125382 28313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "0eddad8efff545069e79723abd6996fb"
format_stamp: "Formatted at 2026-09-27 10:54:16 on dist-test-slave-c60x"
server_key: "fb8e91ee4dce231b80cf67e2b3a9e190"
server_key_iv: "66e95c954ca98b2ce9231c4d7338edfa"
server_key_version: "encryptionkey@0"
I20260927 10:54:16.125957 28313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "0eddad8efff545069e79723abd6996fb"
format_stamp: "Formatted at 2026-09-27 10:54:16 on dist-test-slave-c60x"
server_key: "fb8e91ee4dce231b80cf67e2b3a9e190"
server_key_iv: "66e95c954ca98b2ce9231c4d7338edfa"
server_key_version: "encryptionkey@0"
I20260927 10:54:16.129828 28313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:54:16.132483 28328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:16.133764 28313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:16.133912 28313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "0eddad8efff545069e79723abd6996fb"
format_stamp: "Formatted at 2026-09-27 10:54:16 on dist-test-slave-c60x"
server_key: "fb8e91ee4dce231b80cf67e2b3a9e190"
server_key_iv: "66e95c954ca98b2ce9231c4d7338edfa"
server_key_version: "encryptionkey@0"
I20260927 10:54:16.134037 28313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:16.151562 28313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:16.154779 28313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:16.155010 28313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:16.162714 28313 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:36349
I20260927 10:54:16.162741 28380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:36349 every 8 connection(s)
I20260927 10:54:16.163786 28313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:16.166499 28381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:16.169054 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 28313
I20260927 10:54:16.169200 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:16.169466 6270 external_mini_cluster.cc:1428] Setting key d1a4bbc467e40931aae54dc89983cbba
I20260927 10:54:16.172900 28381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb: Bootstrap starting.
I20260927 10:54:16.175647 28381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:16.176569 28381 log.cc:824] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:16.178990 28381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb: No bootstrap required, opened a new log
I20260927 10:54:16.179710 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:16.170887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60541 (local address 127.6.31.190:36349)
0927 10:54:16.171401 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:16.171419 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:16.171462 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:54:16.172285 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:16.172307 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:16.172742 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:16.173113 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:16.173529 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:16.175008 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:16.176005 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:16.176705 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:16.176856 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:16.177691 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:16.177697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:16.177714 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:16.178215 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:16.178281 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:16.178311 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:16.178411 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:16.178510 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 10:54:16.178739 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":216,"threads_started":1}
I20260927 10:54:16.182555 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:16.183310 28381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eddad8efff545069e79723abd6996fb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36349 } }
I20260927 10:54:16.183553 28381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:16.183631 28381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eddad8efff545069e79723abd6996fb, State: Initialized, Role: FOLLOWER
I20260927 10:54:16.184096 28381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eddad8efff545069e79723abd6996fb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36349 } }
I20260927 10:54:16.184247 28381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:16.184333 28381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:16.184432 28381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:16.185738 28381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eddad8efff545069e79723abd6996fb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36349 } }
I20260927 10:54:16.186087 28381 leader_election.cc:302] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eddad8efff545069e79723abd6996fb; no voters:
I20260927 10:54:16.186409 28381 leader_election.cc:288] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:16.186522 28391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:16.186776 28391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [term 1 LEADER]: Becoming Leader. State: Replica: 0eddad8efff545069e79723abd6996fb, State: Running, Role: LEADER
I20260927 10:54:16.187180 28391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eddad8efff545069e79723abd6996fb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36349 } }
I20260927 10:54:16.187794 28381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:16.189157 28392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eddad8efff545069e79723abd6996fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eddad8efff545069e79723abd6996fb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36349 } } }
I20260927 10:54:16.189297 28392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:16.189466 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:16.184781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48865 (local address 127.6.31.190:36349)
0927 10:54:16.184913 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:16.184918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:16.185114 (+ 196us) server_negotiation.cc:407] Connection header received
0927 10:54:16.185226 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:16.185228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:16.185270 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:16.185330 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:16.185681 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:16.186374 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:16.187485 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:16.187800 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:16.187934 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:16.188890 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:16.188894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:16.188900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:16.188918 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:16.188949 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:16.188963 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:16.189016 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:16.189225 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 10:54:16.189285 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:54:16.189157 28393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eddad8efff545069e79723abd6996fb. Latest consensus state: current_term: 1 leader_uuid: "0eddad8efff545069e79723abd6996fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eddad8efff545069e79723abd6996fb" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 36349 } } }
I20260927 10:54:16.189733 28393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:16.190395 28400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:16.193012 28400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:16.198218 28400 catalog_manager.cc:1347] Generated new cluster ID: 985d210f1bca42fea63efa60d8f8c3c5
I20260927 10:54:16.198297 28400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:16.217798 28400 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:16.218786 28400 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:16.224771 28400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eddad8efff545069e79723abd6996fb: Generated new TSK 0
I20260927 10:54:16.225492 28400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:16.234529 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 28313
2026-09-27T10:54:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:16 dist-test-slave-c60x krb5kdc[28412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:16 dist-test-slave-c60x krb5kdc[28412](info): set up 2 sockets
Sep 27 10:54:16 dist-test-slave-c60x krb5kdc[28412](info): commencing operation
krb5kdc: starting...
W20260927 10:54:17.279422 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:17 dist-test-slave-c60x krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:54:17.410593 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:44687
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:44687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:17.522704 28425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:17.523011 28425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:17.523054 28425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:17.527318 28425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:17.527390 28425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:17.527410 28425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:17.527427 28425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:17.527442 28425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:17.533003 28425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:44687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:44687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:17.534166 28425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:17.535290 28425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:17.543954 28433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:17.543979 28431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:17.544005 28430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:17.544003 28425 server_base.cc:1034] running on GCE node
I20260927 10:54:17.544744 28425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:17.545912 28425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:17.547137 28425 hybrid_clock.cc:648] HybridClock initialized: now 1790506457547052 us; error 45 us; skew 500 ppm
Sep 27 10:54:17 dist-test-slave-c60x krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506457, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:17.550184 28425 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:54:17.551237 28425 webserver.cc:460] Webserver started at http://127.6.31.190:35541/ using document root <none> and password file <none>
I20260927 10:54:17.551849 28425 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:17.551944 28425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:17.552184 28425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:17.554301 28425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "4e3e55e941264ffe99f6c8be29396243"
format_stamp: "Formatted at 2026-09-27 10:54:17 on dist-test-slave-c60x"
server_key: "018594ce576234c92e6a9e311dd63656"
server_key_iv: "95ed80c38eb2089f14a5738da70751a8"
server_key_version: "encryptionkey@0"
I20260927 10:54:17.554880 28425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "4e3e55e941264ffe99f6c8be29396243"
format_stamp: "Formatted at 2026-09-27 10:54:17 on dist-test-slave-c60x"
server_key: "018594ce576234c92e6a9e311dd63656"
server_key_iv: "95ed80c38eb2089f14a5738da70751a8"
server_key_version: "encryptionkey@0"
I20260927 10:54:17.558857 28425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 10:54:17.561364 28440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:17.562669 28425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:17.562832 28425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "4e3e55e941264ffe99f6c8be29396243"
format_stamp: "Formatted at 2026-09-27 10:54:17 on dist-test-slave-c60x"
server_key: "018594ce576234c92e6a9e311dd63656"
server_key_iv: "95ed80c38eb2089f14a5738da70751a8"
server_key_version: "encryptionkey@0"
I20260927 10:54:17.562961 28425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:17.567898 28425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:17.571053 28425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:17.571337 28425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:17.579176 28425 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:44687
I20260927 10:54:17.579360 28492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:44687 every 8 connection(s)
I20260927 10:54:17.580312 28425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:17.583256 28493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:17.587224 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 28425
I20260927 10:54:17.587379 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:17.587666 6270 external_mini_cluster.cc:1428] Setting key 2bafbee47d481ee30440b41b37fc1c7c
I20260927 10:54:17.589690 28493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243: Bootstrap starting.
I20260927 10:54:17.592308 28493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:17.593025 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:17.589122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51549 (local address 127.6.31.190:44687)
0927 10:54:17.589572 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:17.589588 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:17.589623 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:54:17.590311 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:17.590327 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:17.590615 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:17.590894 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:17.591791 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:17.591796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:17.591806 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:17.591814 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:17.591856 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:17.591872 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:17.591959 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:17.592128 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 10:54:17.592316 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":148,"threads_started":1}
I20260927 10:54:17.593048 28493 log.cc:824] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:17.595082 28493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243: No bootstrap required, opened a new log
I20260927 10:54:17.595780 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:17.598917 28493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e55e941264ffe99f6c8be29396243" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44687 } }
I20260927 10:54:17.599243 28493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:17.599308 28493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3e55e941264ffe99f6c8be29396243, State: Initialized, Role: FOLLOWER
I20260927 10:54:17.599658 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:17.597849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45927 (local address 127.6.31.190:44687)
0927 10:54:17.597964 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:17.597970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:17.598177 (+ 207us) server_negotiation.cc:407] Connection header received
0927 10:54:17.598336 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:17.598339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:17.598373 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:17.598462 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:17.599262 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:17.599265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:17.599268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:17.599279 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:17.599309 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:17.599322 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:17.599398 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:17.599479 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 10:54:17.599521 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 10:54:17.599849 28493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e55e941264ffe99f6c8be29396243" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44687 } }
I20260927 10:54:17.600023 28493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:17.600091 28493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:17.600188 28493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:17.601131 28493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e55e941264ffe99f6c8be29396243" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44687 } }
I20260927 10:54:17.601476 28493 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3e55e941264ffe99f6c8be29396243; no voters:
I20260927 10:54:17.601797 28493 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:17.601927 28503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:17.602149 28503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [term 1 LEADER]: Becoming Leader. State: Replica: 4e3e55e941264ffe99f6c8be29396243, State: Running, Role: LEADER
I20260927 10:54:17.602483 28503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e55e941264ffe99f6c8be29396243" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44687 } }
I20260927 10:54:17.603035 28493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:17.604179 28505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3e55e941264ffe99f6c8be29396243. Latest consensus state: current_term: 1 leader_uuid: "4e3e55e941264ffe99f6c8be29396243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e55e941264ffe99f6c8be29396243" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44687 } } }
I20260927 10:54:17.604302 28505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:17.604624 28504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e3e55e941264ffe99f6c8be29396243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e55e941264ffe99f6c8be29396243" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 44687 } } }
I20260927 10:54:17.604662 28512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:17.604748 28504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:17.607955 28512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:17.613598 28512 catalog_manager.cc:1347] Generated new cluster ID: e03b97c923b147b1b22408262a00f1d4
I20260927 10:54:17.613672 28512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:17.634919 28512 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:17.635317 28512 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:17.650452 28512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e3e55e941264ffe99f6c8be29396243: Generated new TSK 0
I20260927 10:54:17.651394 28512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:17.659883 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 28425
2026-09-27T10:54:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1423 ms)
[----------] 18 tests from AuthTokenIssuingTest (26112 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:17 dist-test-slave-c60x krb5kdc[28524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:17 dist-test-slave-c60x krb5kdc[28524](info): set up 2 sockets
Sep 27 10:54:17 dist-test-slave-c60x krb5kdc[28524](info): commencing operation
krb5kdc: starting...
W20260927 10:54:18.719483 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:18 dist-test-slave-c60x krb5kdc[28524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:54:18.847250 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:43093
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:43093,127.6.31.189:33199,127.6.31.188:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:18.957878 28537 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:54:18.958168 28537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:18.958199 28537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:18.958238 28537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:18.962149 28537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:18.962244 28537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:18.962291 28537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:18.962328 28537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:18.962369 28537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:18.967664 28537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:43093,127.6.31.189:33199,127.6.31.188:46845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:43093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:18.968708 28537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:18.969681 28537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:18.978381 28545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:18.978490 28543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:18.978461 28537 server_base.cc:1034] running on GCE node
W20260927 10:54:18.978381 28542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:18.979211 28537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:18.980435 28537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:18.981627 28537 hybrid_clock.cc:648] HybridClock initialized: now 1790506458981612 us; error 27 us; skew 500 ppm
Sep 27 10:54:18 dist-test-slave-c60x krb5kdc[28524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506458, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:18.999750 28537 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:54:19.000826 28537 webserver.cc:460] Webserver started at http://127.6.31.190:33167/ using document root <none> and password file <none>
I20260927 10:54:19.001425 28537 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:19.001516 28537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:19.001765 28537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:19.004015 28537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "af41ef85ef394abead2a9591b8cbbcdf"
format_stamp: "Formatted at 2026-09-27 10:54:18 on dist-test-slave-c60x"
server_key: "72c75abf0f45373198a857657073b11f"
server_key_iv: "ccc2a65e9916fde0e23dedc95f1baa25"
server_key_version: "encryptionkey@0"
I20260927 10:54:19.004598 28537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "af41ef85ef394abead2a9591b8cbbcdf"
format_stamp: "Formatted at 2026-09-27 10:54:18 on dist-test-slave-c60x"
server_key: "72c75abf0f45373198a857657073b11f"
server_key_iv: "ccc2a65e9916fde0e23dedc95f1baa25"
server_key_version: "encryptionkey@0"
I20260927 10:54:19.008369 28537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:54:19.010800 28552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:19.012128 28537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:19.012318 28537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "af41ef85ef394abead2a9591b8cbbcdf"
format_stamp: "Formatted at 2026-09-27 10:54:18 on dist-test-slave-c60x"
server_key: "72c75abf0f45373198a857657073b11f"
server_key_iv: "ccc2a65e9916fde0e23dedc95f1baa25"
server_key_version: "encryptionkey@0"
I20260927 10:54:19.012462 28537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:19.025962 28537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:19.028939 28537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:19.029150 28537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:19.036041 28537 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:43093
I20260927 10:54:19.036336 28604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:43093 every 8 connection(s)
I20260927 10:54:19.037027 28537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:19.039755 28605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:19.043394 28605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:19.043860 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 28537
I20260927 10:54:19.043985 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:19.044309 6270 external_mini_cluster.cc:1428] Setting key 58ed7095256f1d1bb2827d4f5a599b35
WARNING: no policy specified for kudu/127.6.31.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.189@KRBTEST.COM" created.
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506458, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:54:19.061750 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.045676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43093 (local address 127.0.0.1:50630)
0927 10:54:19.046241 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.046294 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:19.047327 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:19.048054 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:19.048062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:19.048652 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:19.049042 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.049065 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.050224 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.050230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.051524 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.051535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.052206 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.052213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.052374 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.052967 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:19.052994 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:19.056711 (+ 3717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:19.058850 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.058858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.058866 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.059151 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.059550 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.059552 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.059553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.059659 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.060445 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:19.060450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:19.060570 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 10:54:19.060707 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 10:54:19.060972 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":145,"threads_started":1}
I20260927 10:54:19.061754 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.045681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.6.31.190:43093)
0927 10:54:19.046262 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.046281 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.046409 (+ 128us) server_negotiation.cc:407] Connection header received
0927 10:54:19.047641 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.047646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.047829 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.047984 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.049239 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.050087 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.051654 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.052028 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.052142 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.056873 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.056879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.056890 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.056910 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.058732 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.059300 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.059309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.059314 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.059455 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.059765 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.059772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.059776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.060332 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.060482 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.060846 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.061088 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":460,"spinlock_wait_cycles":33152,"thread_start_us":203,"threads_started":1}
I20260927 10:54:19.064116 28605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:19.064831 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.064585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33199 (local address 127.0.0.1:55324)
0927 10:54:19.064701 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.064775 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:54:19.065042 28555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
I20260927 10:54:19.065881 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.065680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33199 (local address 127.0.0.1:55330)
0927 10:54:19.065794 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.065835 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 10:54:19.066016 28605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.189:33199: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab.
Entry for principal kudu/127.6.31.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab.
I20260927 10:54:19.074419 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.189:33199
--webserver_interface=127.6.31.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab
--principal=kudu/127.6.31.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:43093,127.6.31.189:33199,127.6.31.188:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:54:19.115279 28605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } attempt: 1
I20260927 10:54:19.116307 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.115932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33199 (local address 127.0.0.1:55344)
0927 10:54:19.116169 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.116230 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 10:54:19.117031 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.116815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33199 (local address 127.0.0.1:55348)
0927 10:54:19.116922 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.116981 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:54:19.117163 28605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.189:33199: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
I20260927 10:54:19.185348 28605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } attempt: 2
I20260927 10:54:19.186220 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.185920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33199 (local address 127.0.0.1:55350)
0927 10:54:19.186069 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.186136 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 10:54:19.186971 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.186755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33199 (local address 127.0.0.1:55366)
0927 10:54:19.186861 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.186910 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:54:19.187144 28605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.189:33199: Network error: Client connection negotiation failed: client connection to 127.6.31.189:33199: connect: Connection refused (error 111)
W20260927 10:54:19.186978 28612 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:54:19.187407 28612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:19.187477 28612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:19.187574 28612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:19.191982 28612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:19.192049 28612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:19.192070 28612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:19.192086 28612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:19.192123 28612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:19.197974 28612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:43093,127.6.31.189:33199,127.6.31.188:46845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab
--principal=kudu/127.6.31.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.189:33199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.31.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:19.199292 28612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:19.200376 28612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:19.209008 28620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:19.209146 28612 server_base.cc:1034] running on GCE node
W20260927 10:54:19.209033 28617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:19.209209 28618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:19.209900 28612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:19.211057 28612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:19.212286 28612 hybrid_clock.cc:648] HybridClock initialized: now 1790506459212273 us; error 27 us; skew 500 ppm
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506459, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:19.215046 28612 init.cc:377] Logged in from keytab as kudu/127.6.31.189@KRBTEST.COM (short username kudu)
I20260927 10:54:19.216079 28612 webserver.cc:460] Webserver started at http://127.6.31.189:36961/ using document root <none> and password file <none>
I20260927 10:54:19.216763 28612 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:19.216856 28612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:19.217082 28612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:19.219195 28612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/data/instance:
uuid: "526b118611214448b8dcaf01b8a92c2f"
format_stamp: "Formatted at 2026-09-27 10:54:19 on dist-test-slave-c60x"
server_key: "782803a2b6ea327a49c422cc9707da7f"
server_key_iv: "6c73d3012588478256272bb3d2857bfb"
server_key_version: "encryptionkey@0"
I20260927 10:54:19.219766 28612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/wal/instance:
uuid: "526b118611214448b8dcaf01b8a92c2f"
format_stamp: "Formatted at 2026-09-27 10:54:19 on dist-test-slave-c60x"
server_key: "782803a2b6ea327a49c422cc9707da7f"
server_key_iv: "6c73d3012588478256272bb3d2857bfb"
server_key_version: "encryptionkey@0"
I20260927 10:54:19.223491 28612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:54:19.225871 28627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:19.227176 28612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:19.227311 28612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/wal
uuid: "526b118611214448b8dcaf01b8a92c2f"
format_stamp: "Formatted at 2026-09-27 10:54:19 on dist-test-slave-c60x"
server_key: "782803a2b6ea327a49c422cc9707da7f"
server_key_iv: "6c73d3012588478256272bb3d2857bfb"
server_key_version: "encryptionkey@0"
I20260927 10:54:19.227425 28612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:19.256646 28612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:19.259794 28612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:19.259959 28612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:19.267068 28612 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.189:33199
I20260927 10:54:19.267434 28679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.189:33199 every 8 connection(s)
I20260927 10:54:19.268106 28612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
I20260927 10:54:19.271021 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 28612
I20260927 10:54:19.271139 28680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:19.271200 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-1/wal/instance
I20260927 10:54:19.271503 6270 external_mini_cluster.cc:1428] Setting key 520229889cc0185063ee08e6bd2df055
I20260927 10:54:19.275395 28680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:19.278312 28605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } attempt: 3
WARNING: no policy specified for kudu/127.6.31.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.188@KRBTEST.COM" created.
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506458, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506459, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:54:19.291908 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.278749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33199 (local address 127.0.0.1:55370)
0927 10:54:19.278924 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.278940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:19.279034 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:19.280072 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:19.280075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:19.280261 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:19.280372 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.280380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.282378 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.282382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.283228 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.283239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.284115 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.284122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.284237 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.284882 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:19.284900 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:19.286878 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:19.289678 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.289684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.289689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.290054 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.290407 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.290413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.290418 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.290525 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.291336 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:19.291339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:19.291438 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 10:54:19.291567 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 10:54:19.291718 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 10:54:19.292295 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.277359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.6.31.190:43093)
0927 10:54:19.277530 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.277535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.278457 (+ 922us) server_negotiation.cc:407] Connection header received
0927 10:54:19.279867 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.279870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.279914 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.280080 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.281239 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.282022 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.283552 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.283793 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.283876 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.287356 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.287361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.287367 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.287398 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.289870 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.290661 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.290664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.290666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.290744 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.291031 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.291034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.291036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.291324 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.291428 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.291971 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.292106 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:54:19.293099 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.278801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.6.31.189:33199)
0927 10:54:19.279242 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.279251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.279312 (+ 61us) server_negotiation.cc:407] Connection header received
0927 10:54:19.279650 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.279671 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.279897 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.280536 (+ 639us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.280889 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.282277 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.283385 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.283889 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.284043 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.287040 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.287047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.287067 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.287132 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.289533 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.290179 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.290187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.290191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.290308 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.290622 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.290624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.290626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.291229 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.291369 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.291696 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.292047 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":358,"spinlock_wait_cycles":21760,"thread_start_us":130,"threads_started":1}
I20260927 10:54:19.293103 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.277735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43093 (local address 127.0.0.1:50642)
0927 10:54:19.278291 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.278352 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:19.279646 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:19.280081 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:19.280092 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:19.280573 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:19.281076 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.281094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.282150 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.282154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.283444 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.283452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.283947 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.283954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.284064 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.284739 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:19.284763 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:19.287201 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:19.289976 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.289986 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.289997 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.290302 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.290838 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.290841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.290843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.290919 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.291489 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:19.291499 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:19.291687 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 10:54:19.292508 (+ 821us) client_negotiation.cc:240] Negotiation successful
0927 10:54:19.292748 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":23,"spinlock_wait_cycles":5504,"thread_start_us":114,"threads_started":1}
I20260927 10:54:19.294487 28680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:19.295491 28605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:19.296235 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.295939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34796)
0927 10:54:19.296080 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.296165 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 10:54:19.296934 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.296633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34808)
0927 10:54:19.296779 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.296840 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 10:54:19.297114 28605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:46845: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506459, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab.
Entry for principal kudu/127.6.31.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab.
I20260927 10:54:19.302599 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.188:46845
--webserver_interface=127.6.31.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab
--principal=kudu/127.6.31.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:43093,127.6.31.189:33199,127.6.31.188:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:54:19.304376 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.295194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.6.31.189:33199)
0927 10:54:19.295294 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.295298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.295312 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:54:19.295390 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.295393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.295439 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.295542 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.296014 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.296756 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.297973 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.298491 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.298663 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.300816 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.300822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.300824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.300851 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.302586 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.303065 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.303071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.303076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.303184 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.303447 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.303453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.303458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.303704 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.303807 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.304063 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.304251 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":39}
I20260927 10:54:19.304400 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.295020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33199 (local address 127.0.0.1:55384)
0927 10:54:19.295229 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.295244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:19.295322 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:19.295563 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:19.295570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:19.295774 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:19.295897 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.295905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.296880 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.296883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.297650 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.297659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.298685 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.298696 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.298769 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.299324 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:19.299335 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:19.300629 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:19.302725 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.302730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.302733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.302966 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.303283 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.303285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.303287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.303361 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.303796 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:19.303799 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:19.303869 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 10:54:19.303985 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:54:19.304187 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 10:54:19.305411 28680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:19.306164 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.305843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34818)
0927 10:54:19.305980 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.306075 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 10:54:19.306393 28631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
I20260927 10:54:19.307391 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.307073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34834)
0927 10:54:19.307252 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.307315 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 10:54:19.307606 28680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:46845: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
I20260927 10:54:19.328264 28605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } attempt: 1
I20260927 10:54:19.329130 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.328834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34836)
0927 10:54:19.328978 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.329054 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 10:54:19.330006 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.329721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34852)
0927 10:54:19.329900 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.329949 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260927 10:54:19.330247 28605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:46845: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
I20260927 10:54:19.356938 28680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } attempt: 1
I20260927 10:54:19.357988 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.357688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34866)
0927 10:54:19.357836 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.357908 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 10:54:19.359156 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.358869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34876)
0927 10:54:19.358981 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.359038 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:54:19.359319 28680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:46845: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
I20260927 10:54:19.405439 28605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } attempt: 2
I20260927 10:54:19.406282 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.405989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34892)
0927 10:54:19.406150 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.406208 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 10:54:19.407061 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.406827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34902)
0927 10:54:19.406922 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.406963 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:54:19.407315 28605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:46845: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
W20260927 10:54:19.418227 28686 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:54:19.418517 28686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:19.418574 28686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:19.418665 28686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:19.422865 28686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:19.422940 28686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:19.422959 28686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:19.422976 28686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:19.422991 28686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:19.427507 28680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } attempt: 2
I20260927 10:54:19.428428 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.428150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34904)
0927 10:54:19.428283 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.428353 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:54:19.428751 28686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:43093,127.6.31.189:33199,127.6.31.188:46845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab
--principal=kudu/127.6.31.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.188:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.31.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:19.429204 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.428957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34916)
0927 10:54:19.429086 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.429140 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:54:19.429360 28680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:46845: Network error: Client connection negotiation failed: client connection to 127.6.31.188:46845: connect: Connection refused (error 111)
I20260927 10:54:19.429977 28686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:19.431041 28686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:54:19.439935 28686 server_base.cc:1034] running on GCE node
W20260927 10:54:19.439884 28692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:19.439893 28695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:19.439884 28693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:19.440788 28686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:19.441959 28686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:19.443179 28686 hybrid_clock.cc:648] HybridClock initialized: now 1790506459443152 us; error 42 us; skew 500 ppm
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506459, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:19.446055 28686 init.cc:377] Logged in from keytab as kudu/127.6.31.188@KRBTEST.COM (short username kudu)
I20260927 10:54:19.446981 28686 webserver.cc:460] Webserver started at http://127.6.31.188:45305/ using document root <none> and password file <none>
I20260927 10:54:19.447592 28686 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:19.447690 28686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:19.447924 28686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:19.450104 28686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/data/instance:
uuid: "d2fc65e2ccf2460ca3e7297172ff82f3"
format_stamp: "Formatted at 2026-09-27 10:54:19 on dist-test-slave-c60x"
server_key: "d4a2ee48bdede975946286489c9f622b"
server_key_iv: "11dfe259d0d6a887333b70c23423ab80"
server_key_version: "encryptionkey@0"
I20260927 10:54:19.450666 28686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/wal/instance:
uuid: "d2fc65e2ccf2460ca3e7297172ff82f3"
format_stamp: "Formatted at 2026-09-27 10:54:19 on dist-test-slave-c60x"
server_key: "d4a2ee48bdede975946286489c9f622b"
server_key_iv: "11dfe259d0d6a887333b70c23423ab80"
server_key_version: "encryptionkey@0"
I20260927 10:54:19.454414 28686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:54:19.456848 28702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:19.458105 28686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:54:19.458282 28686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/wal
uuid: "d2fc65e2ccf2460ca3e7297172ff82f3"
format_stamp: "Formatted at 2026-09-27 10:54:19 on dist-test-slave-c60x"
server_key: "d4a2ee48bdede975946286489c9f622b"
server_key_iv: "11dfe259d0d6a887333b70c23423ab80"
server_key_version: "encryptionkey@0"
I20260927 10:54:19.458403 28686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:19.477782 28686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:19.481060 28686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:19.481273 28686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:19.489272 28686 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.188:46845
I20260927 10:54:19.489552 28754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.188:46845 every 8 connection(s)
I20260927 10:54:19.490388 28686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
I20260927 10:54:19.493184 28755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:19.496944 28755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:19.500270 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 28686
I20260927 10:54:19.500427 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506175715859-6270-0/minicluster-data/master-2/wal/instance
I20260927 10:54:19.500691 6270 external_mini_cluster.cc:1428] Setting key fe88c46297c7c35fbe48ac62b6b54801
I20260927 10:54:19.506481 28605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } attempt: 3
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506459, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
I20260927 10:54:19.512468 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.498809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.6.31.190:43093)
0927 10:54:19.498965 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.498970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.499881 (+ 911us) server_negotiation.cc:407] Connection header received
0927 10:54:19.500872 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.500875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.500940 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.501038 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.502504 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.503281 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.504878 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.505191 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.505303 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.508209 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.508213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.508217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.508244 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.510065 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.510702 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.510705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.510707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.510796 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.511244 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.511248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.511249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.511509 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.511600 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.512178 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.512304 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506458, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
I20260927 10:54:19.513499 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.499272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:43093 (local address 127.0.0.1:50644)
0927 10:54:19.499730 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.499772 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:19.500645 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:19.501277 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:19.501285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:19.501765 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:19.502366 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.502380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.503433 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.503437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.504742 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.504753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.505341 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.505343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.505539 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.506152 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:19.506188 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:19.508090 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:19.510174 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.510181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.510192 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.510592 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.510879 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.510883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.510885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.511144 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.511605 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:19.511615 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:19.511873 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 10:54:19.512089 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 10:54:19.512430 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":184,"thread_start_us":120,"threads_started":1}
I20260927 10:54:19.514111 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.501414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.6.31.188:46845)
0927 10:54:19.501778 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.501789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.501811 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:54:19.501851 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.501856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.502024 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.502177 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.502940 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.503934 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.505121 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.505614 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.505700 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.508627 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.508635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.508650 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.508678 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.510727 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.511326 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.511332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.511333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.511410 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.512177 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.512180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.512183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.512461 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.512674 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.513198 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.513340 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":272,"thread_start_us":134,"threads_started":1}
I20260927 10:54:19.514847 28755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:19.518682 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.506926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34932)
0927 10:54:19.507129 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.507145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:19.507227 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:19.507547 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:19.507550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:19.507738 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:19.507829 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.507835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.508908 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.508913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.509748 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.509759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.510526 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.510533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.510624 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.511552 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:19.511564 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:19.512931 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:19.517141 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.517145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.517147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.517393 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.517705 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.517709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.517710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.517797 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.518188 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:19.518191 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:19.518283 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 10:54:19.518392 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:54:19.518527 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 10:54:19.519032 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.506966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.6.31.188:46845)
0927 10:54:19.507350 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.507358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.507368 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:54:19.507394 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.507396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.507426 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.507523 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.507955 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.508782 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.509857 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.510269 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.510378 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.513077 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.513080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.513082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.513118 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.517034 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.517500 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.517503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.517505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.517599 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.517899 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.517902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.517904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.518080 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.518180 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.518458 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.518566 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":168,"threads_started":1}
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506459, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:54:19.522609 28605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf: Bootstrap starting.
I20260927 10:54:19.524042 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.515381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:33199 (local address 127.0.0.1:55396)
0927 10:54:19.515513 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.515526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:19.515609 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:19.516047 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:19.516053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:19.516227 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:19.516321 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.516328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.517350 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.517355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.518168 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.518175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.518815 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.518821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.518905 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.519449 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:19.519462 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:19.520708 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:19.522447 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.522451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.522454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.522730 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.523049 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.523053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.523055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.523186 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.523564 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:19.523567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:19.523656 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 10:54:19.523765 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:54:19.523889 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 10:54:19.524083 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.515464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.6.31.189:33199)
0927 10:54:19.515572 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.515576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.515592 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:54:19.515766 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.515770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.515829 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.515906 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.516429 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.517233 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.518254 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.518597 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.518695 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.520842 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.520846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.520849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.520874 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.522290 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.522822 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.522825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.522827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.522941 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.523282 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.523285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.523286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.523452 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.523573 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.523828 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.523935 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:54:19.524931 28755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:19.525678 28605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:19.525990 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.516230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.6.31.190:43093)
0927 10:54:19.516338 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.516342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.516407 (+ 65us) server_negotiation.cc:407] Connection header received
0927 10:54:19.516570 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.516573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.516613 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.516659 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.517448 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.517919 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.518968 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.519355 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.519490 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.522170 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.522173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.522176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.522199 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.523887 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.524395 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.524398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.524399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.524477 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.524799 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.524803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.524805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.524991 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.525180 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.525781 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.525889 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:54:19.526684 28605 log.cc:824] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:19.527051 28680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } attempt: 3
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506459, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
I20260927 10:54:19.530297 28605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf: No bootstrap required, opened a new log
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506459, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
Sep 27 10:54:19 dist-test-slave-c60x krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
I20260927 10:54:19.534278 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.525451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34942 (local address 127.6.31.188:46845)
0927 10:54:19.525572 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.525576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.525592 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:54:19.525738 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.525741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.525776 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.525837 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.526326 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.527037 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.528137 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.528478 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.528577 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.530709 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.530712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.530715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.530736 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.532500 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.532941 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.532944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.532946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.533013 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.533261 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.533264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.533265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.533499 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.533599 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.533905 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.534085 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":50}
I20260927 10:54:19.534878 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.525362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34942)
0927 10:54:19.525543 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.525559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:19.525635 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:19.525942 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:19.525947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:19.526114 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:19.526211 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.526217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.527255 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.527258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.528009 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.528021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.528671 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.528674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.528757 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.529198 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:19.529212 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:19.530569 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:19.532612 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.532615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.532617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.532843 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.533091 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.533094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.533096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.533179 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.533620 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:19.533625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:19.533721 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 10:54:19.533837 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:54:19.534152 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":90}
I20260927 10:54:19.534825 28605 raft_consensus.cc:348] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } }
I20260927 10:54:19.535089 28605 raft_consensus.cc:729] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af41ef85ef394abead2a9591b8cbbcdf, State: Initialized, Role: FOLLOWER
I20260927 10:54:19.535832 28605 consensus_queue.cc:260] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } }
I20260927 10:54:19.536404 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.536638 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:19.537348 28605 sys_catalog.cc:564] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:19.538405 28755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3: Bootstrap starting.
I20260927 10:54:19.538568 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.527014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.6.31.189:33199)
0927 10:54:19.527155 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.527159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.527321 (+ 162us) server_negotiation.cc:407] Connection header received
0927 10:54:19.527507 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.527510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.527560 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.527612 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.528336 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.528853 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.530159 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.530496 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.530581 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.535235 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.535239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.535241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.535268 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.536949 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.537380 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.537385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.537387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.537471 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.537751 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.537755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.537757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.537935 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.538142 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.538284 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.538394 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":41}
I20260927 10:54:19.542040 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 10:54:19.543036 28773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:19.543190 28773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:19.543933 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.527494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:46845 (local address 127.0.0.1:34946)
0927 10:54:19.527707 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:19.527720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:19.527786 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:19.528417 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:19.528420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:19.528594 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:19.528694 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.528700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.529677 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.529680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.530751 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:19.530756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.531479 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.531485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:19.531583 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.532026 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:19.532043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:19.534167 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:19.542188 (+ 8021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.542191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.542194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.542431 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.542754 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:19.542758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:19.542760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:19.542844 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:19.543386 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:19.543392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:19.543495 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 10:54:19.543592 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:54:19.543760 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":294}
I20260927 10:54:19.543857 28579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "af41ef85ef394abead2a9591b8cbbcdf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50660
I20260927 10:54:19.544090 28579 raft_consensus.cc:482] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:54:19.544205 28579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:19.544600 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.527898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.6.31.188:46845)
0927 10:54:19.528151 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.528155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.528192 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:54:19.528219 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.528221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.528256 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.528316 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.528811 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.529556 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.530854 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.531274 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.531365 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.534307 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.534313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.534316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.534338 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.542055 (+ 7717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.542561 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.542564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.542566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.542654 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.542962 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.542966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.542968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.543273 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.544221 (+ 948us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.544304 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.544430 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:54:19.545118 28755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:19.542205 28537 master_runner.cc:186] Error getting master registration for 127.6.31.189:33199: OK, instance_id { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" instance_seqno: 1790506459265109 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:54:19.545624 28537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:54:19.545701 28579 raft_consensus.cc:504] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } }
I20260927 10:54:19.546001 28755 log.cc:824] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:19.546705 28579 leader_election.cc:288] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [CANDIDATE]: Term 1 election: Requested vote from peers 526b118611214448b8dcaf01b8a92c2f (127.6.31.189:33199), d2fc65e2ccf2460ca3e7297172ff82f3 (127.6.31.188:46845)
I20260927 10:54:19.547576 28680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f: Bootstrap starting.
I20260927 10:54:19.548426 28755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3: No bootstrap required, opened a new log
I20260927 10:54:19.550231 28680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:19.551040 28680 log.cc:824] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:19.552515 28755 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } }
I20260927 10:54:19.552754 28755 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fc65e2ccf2460ca3e7297172ff82f3, State: Initialized, Role: FOLLOWER
I20260927 10:54:19.553061 28755 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } }
I20260927 10:54:19.553519 28781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.553701 28781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:19.554302 28755 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:19.554344 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af41ef85ef394abead2a9591b8cbbcdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3"
I20260927 10:54:19.554572 28729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:19.555969 28729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af41ef85ef394abead2a9591b8cbbcdf in term 1.
I20260927 10:54:19.556555 28556 leader_election.cc:302] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af41ef85ef394abead2a9591b8cbbcdf, d2fc65e2ccf2460ca3e7297172ff82f3; no voters:
I20260927 10:54:19.556648 28680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f: No bootstrap required, opened a new log
I20260927 10:54:19.556929 28762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:19.557190 28762 raft_consensus.cc:686] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [term 1 LEADER]: Becoming Leader. State: Replica: af41ef85ef394abead2a9591b8cbbcdf, State: Running, Role: LEADER
I20260927 10:54:19.557549 28762 consensus_queue.cc:237] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } }
W20260927 10:54:19.560549 28792 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:19.560640 28792 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:19.561488 28793 sys_catalog.cc:455] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader af41ef85ef394abead2a9591b8cbbcdf. Latest consensus state: current_term: 1 leader_uuid: "af41ef85ef394abead2a9591b8cbbcdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.561740 28793 sys_catalog.cc:458] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:19.563410 28686 master_runner.cc:186] Error getting master registration for 127.6.31.189:33199: OK, instance_id { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" instance_seqno: 1790506459265109 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:54:19.566751 28686 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:54:19.566582 28680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } }
I20260927 10:54:19.566820 28680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526b118611214448b8dcaf01b8a92c2f, State: Initialized, Role: FOLLOWER
I20260927 10:54:19.567010 28796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:19.567206 28680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } }
I20260927 10:54:19.567785 28799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.568063 28799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:19.568722 28680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:19.568758 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af41ef85ef394abead2a9591b8cbbcdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "526b118611214448b8dcaf01b8a92c2f"
I20260927 10:54:19.568907 28796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:19.568979 28654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:19.570401 28654 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af41ef85ef394abead2a9591b8cbbcdf in term 1.
I20260927 10:54:19.576717 28654 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [term 1 FOLLOWER]: Refusing update from remote peer af41ef85ef394abead2a9591b8cbbcdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:19.577505 28793 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:19.578073 28729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [term 1 FOLLOWER]: Refusing update from remote peer af41ef85ef394abead2a9591b8cbbcdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:19.580476 28793 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:19.584635 28781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af41ef85ef394abead2a9591b8cbbcdf. Latest consensus state: current_term: 1 leader_uuid: "af41ef85ef394abead2a9591b8cbbcdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.584772 28781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:19.585541 28799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader af41ef85ef394abead2a9591b8cbbcdf. Latest consensus state: current_term: 1 leader_uuid: "af41ef85ef394abead2a9591b8cbbcdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.585672 28799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [sys.catalog]: This master's current role is: FOLLOWER
W20260927 10:54:19.587375 28811 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:19.587505 28811 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:19.592567 28793 sys_catalog.cc:455] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af41ef85ef394abead2a9591b8cbbcdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.592733 28793 sys_catalog.cc:458] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:19.592880 28812 mvcc.cc:205] Tried to move back new op lower bound from 7333914458409627648 to 7333914458351398912. Current Snapshot: MvccSnapshot[applied={T|T < 7333914458409627648}]
I20260927 10:54:19.594667 28781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af41ef85ef394abead2a9591b8cbbcdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.594815 28781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:19.595763 28816 mvcc.cc:205] Tried to move back new op lower bound from 7333914458409627648 to 7333914458351398912. Current Snapshot: MvccSnapshot[applied={T|T < 7333914458409627648}]
I20260927 10:54:19.599630 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.562347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.6.31.189:33199)
0927 10:54:19.562456 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.562462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.575623 (+ 13161us) server_negotiation.cc:407] Connection header received
0927 10:54:19.578648 (+ 3025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.578653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.578708 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.578763 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.580056 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.580741 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.584565 (+ 3824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.584996 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.585105 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.593487 (+ 8382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.593491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.593494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.593527 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.595323 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.598132 (+ 2809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.598136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.598138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.598238 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.598570 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.598574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.598575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.598845 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.598963 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.599349 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.599479 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:54:19.604001 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af41ef85ef394abead2a9591b8cbbcdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.604173 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:19.605562 28799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af41ef85ef394abead2a9591b8cbbcdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af41ef85ef394abead2a9591b8cbbcdf" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 43093 } } peers { permanent_uuid: "526b118611214448b8dcaf01b8a92c2f" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 33199 } } peers { permanent_uuid: "d2fc65e2ccf2460ca3e7297172ff82f3" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 46845 } } }
I20260927 10:54:19.605710 28799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:19.607141 28814 mvcc.cc:205] Tried to move back new op lower bound from 7333914458409627648 to 7333914458351398912. Current Snapshot: MvccSnapshot[applied={T|T < 7333914458409627648}]
I20260927 10:54:19.609064 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.567773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.6.31.188:46845)
0927 10:54:19.567898 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.567903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.582258 (+ 14355us) server_negotiation.cc:407] Connection header received
0927 10:54:19.582513 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.582516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.582562 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.582644 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.585316 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.586078 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.591014 (+ 4936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.591352 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.591460 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.601360 (+ 9900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.601364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.601367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.601400 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.603275 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.606233 (+ 2958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.606255 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.606257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.606366 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.607202 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.607206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.607208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.607464 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.607589 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.608730 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.608880 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:54:19.611126 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:19.561872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.6.31.190:43093)
0927 10:54:19.561992 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.561999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.582079 (+ 20080us) server_negotiation.cc:407] Connection header received
0927 10:54:19.582305 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.582308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.582360 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.582412 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.587922 (+ 5510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.588628 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.592297 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:19.592635 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:19.592729 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:19.603877 (+ 11148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:19.603881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:19.603884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:19.603918 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:19.605604 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.606727 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.606734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.606736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.606830 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:19.607936 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:19.607940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:19.607941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:19.608225 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:19.608358 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:19.610803 (+ 2445us) server_negotiation.cc:299] Negotiation successful
0927 10:54:19.610937 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:54:19.619537 28796 catalog_manager.cc:1347] Generated new cluster ID: 9a214910c9cc472d9cbd649b9233e951
I20260927 10:54:19.619638 28796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:19.653438 28796 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:19.656173 28796 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:19.672153 28796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af41ef85ef394abead2a9591b8cbbcdf: Generated new TSK 0
I20260927 10:54:19.673187 28796 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:54:19.715047 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.711834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.6.31.189:33199)
0927 10:54:19.711969 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.711974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.713881 (+ 1907us) server_negotiation.cc:407] Connection header received
0927 10:54:19.714103 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.714107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.714162 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.714216 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.714898 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55406: BlockingRecv error: recv got EOF from 127.0.0.1:55406 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 10:54:19.716849 28609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.712292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.6.31.190:43093)
0927 10:54:19.712411 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.712416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.715845 (+ 3429us) server_negotiation.cc:407] Connection header received
0927 10:54:19.716018 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.716021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.716075 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.716128 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.716722 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50686: BlockingRecv error: recv got EOF from 127.0.0.1:50686 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:54:19.718483 28759 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.713264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.6.31.188:46845)
0927 10:54:19.713378 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.713383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.717603 (+ 4220us) server_negotiation.cc:407] Connection header received
0927 10:54:19.717778 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.717782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.717832 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.717880 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.718366 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34962: BlockingRecv error: recv got EOF from 127.0.0.1:34962 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 10:54:19.723388 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.721735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.6.31.189:33199)
0927 10:54:19.721848 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.721853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.722338 (+ 485us) server_negotiation.cc:407] Connection header received
0927 10:54:19.722570 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.722574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.722653 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.722707 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.723276 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55414: BlockingRecv error: recv got EOF from 127.0.0.1:55414 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 10:54:19.729138 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.727463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.6.31.189:33199)
0927 10:54:19.727589 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.727593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.728150 (+ 557us) server_negotiation.cc:407] Connection header received
0927 10:54:19.728359 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.728363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.728418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.728472 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.729019 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55416: BlockingRecv error: recv got EOF from 127.0.0.1:55416 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 10:54:19.736423 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.733628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.6.31.189:33199)
0927 10:54:19.733735 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.733739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.735447 (+ 1708us) server_negotiation.cc:407] Connection header received
0927 10:54:19.735666 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.735670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.735724 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.735778 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.736307 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55418: BlockingRecv error: recv got EOF from 127.0.0.1:55418 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 10:54:19.744930 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.743158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.6.31.189:33199)
0927 10:54:19.743314 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.743318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.743909 (+ 591us) server_negotiation.cc:407] Connection header received
0927 10:54:19.744122 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.744126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.744181 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.744235 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.744807 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55424: BlockingRecv error: recv got EOF from 127.0.0.1:55424 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 10:54:19.757742 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.755765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.6.31.189:33199)
0927 10:54:19.755896 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.755901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.756629 (+ 728us) server_negotiation.cc:407] Connection header received
0927 10:54:19.756869 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.756874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.756933 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.757012 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.757614 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55436: BlockingRecv error: recv got EOF from 127.0.0.1:55436 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 10:54:19.778271 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.776874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.6.31.189:33199)
0927 10:54:19.777043 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.777050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.777358 (+ 308us) server_negotiation.cc:407] Connection header received
0927 10:54:19.777488 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.777492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.777557 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.777659 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.778146 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55452: BlockingRecv error: recv got EOF from 127.0.0.1:55452 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 10:54:19.813372 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.812195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.6.31.189:33199)
0927 10:54:19.812357 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.812361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.812473 (+ 112us) server_negotiation.cc:407] Connection header received
0927 10:54:19.812655 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.812658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.812714 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.812807 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.813252 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55468: BlockingRecv error: recv got EOF from 127.0.0.1:55468 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 10:54:19.880575 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:19.879496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.6.31.189:33199)
0927 10:54:19.879662 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:19.879666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:19.879740 (+ 74us) server_negotiation.cc:407] Connection header received
0927 10:54:19.879871 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:19.879873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:19.879920 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:19.879992 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:19.880450 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55472: BlockingRecv error: recv got EOF from 127.0.0.1:55472 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 10:54:20.011706 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:20.010623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.6.31.189:33199)
0927 10:54:20.010785 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:20.010788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:20.010842 (+ 54us) server_negotiation.cc:407] Connection header received
0927 10:54:20.010969 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:20.010971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:20.011018 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:20.011117 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:20.011536 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55480: BlockingRecv error: recv got EOF from 127.0.0.1:55480 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 10:54:20.271209 28684 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:20.269883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.6.31.189:33199)
0927 10:54:20.270015 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:20.270019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:20.270155 (+ 136us) server_negotiation.cc:407] Connection header received
0927 10:54:20.270316 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:20.270318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:20.270365 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:20.270443 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:20.271033 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55490: BlockingRecv error: recv got EOF from 127.0.0.1:55490 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:54:20.562047 28792 catalog_manager.cc:1259] Loaded cluster ID: 9a214910c9cc472d9cbd649b9233e951
I20260927 10:54:20.562142 28792 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3: loading cluster ID for follower catalog manager: success
I20260927 10:54:20.563970 28792 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3: acquiring CA information for follower catalog manager: success
I20260927 10:54:20.565078 28792 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2fc65e2ccf2460ca3e7297172ff82f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:54:20.588862 28811 catalog_manager.cc:1259] Loaded cluster ID: 9a214910c9cc472d9cbd649b9233e951
I20260927 10:54:20.588953 28811 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f: loading cluster ID for follower catalog manager: success
I20260927 10:54:20.590610 28811 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f: acquiring CA information for follower catalog manager: success
I20260927 10:54:20.591886 28811 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 526b118611214448b8dcaf01b8a92c2f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:54:20.788957 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:20.785187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.6.31.189:33199)
0927 10:54:20.785425 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:20.785429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:20.785443 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:54:20.785536 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:20.785538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:20.785661 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:20.785765 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:20.786046 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.786558 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.787454 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.787679 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.787757 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:20.788060 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:20.788499 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:20.788588 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:20.788693 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 10:54:20.788743 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260927 10:54:20.795692 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:20.792252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.6.31.188:46845)
0927 10:54:20.792499 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:20.792503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:20.792516 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:54:20.792594 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:20.792596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:20.792702 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:20.792799 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:20.792970 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.793523 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.794299 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.794516 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.794595 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:20.794836 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:20.795306 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:20.795396 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:20.795461 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 10:54:20.795507 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":111,"threads_started":1}
I20260927 10:54:20.802605 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:20.798948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.6.31.188:46845)
0927 10:54:20.799070 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:20.799075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:20.799338 (+ 263us) server_negotiation.cc:407] Connection header received
0927 10:54:20.799447 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:20.799450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:20.799525 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:20.799600 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:20.799888 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.800635 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.801545 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.801777 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.801892 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:20.802142 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:20.802219 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:20.802298 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:20.802408 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 10:54:20.802464 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:54:20.803241 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:20.798578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.6.31.189:33199)
0927 10:54:20.798722 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:20.798729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:20.799243 (+ 514us) server_negotiation.cc:407] Connection header received
0927 10:54:20.799634 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:20.799637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:20.799691 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:20.799766 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:20.800066 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.800569 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.801957 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.802306 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.802385 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:20.802768 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:20.802841 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:20.802883 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:20.803037 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 10:54:20.803119 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:54:20.803778 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:20.798540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.6.31.190:43093)
0927 10:54:20.799065 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:20.799070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:20.799105 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:54:20.799213 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:20.799216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:20.799381 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:20.799500 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:20.799739 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.800590 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.801957 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:20.802306 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:20.802401 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:20.802729 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:20.803123 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:20.803447 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:20.803537 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:54:20.803590 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":84,"threads_started":1}
I20260927 10:54:20.807355 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 28537
I20260927 10:54:20.814249 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 28612
I20260927 10:54:20.821061 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 28686
2026-09-27T10:54:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3163 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:20 dist-test-slave-c60x krb5kdc[28903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:20 dist-test-slave-c60x krb5kdc[28903](info): set up 2 sockets
Sep 27 10:54:20 dist-test-slave-c60x krb5kdc[28903](info): commencing operation
krb5kdc: starting...
W20260927 10:54:21.863483 6270 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/test-user.keytab.
Sep 27 10:54:21 dist-test-slave-c60x krb5kdc[28903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.31.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
Entry for principal kudu/127.6.31.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab.
I20260927 10:54:22.005481 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38099
--webserver_interface=127.6.31.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:38413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:38099,127.6.31.189:41981,127.6.31.188:45555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:22.117728 28916 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:54:22.118028 28916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:22.118052 28916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:22.118090 28916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:22.122192 28916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:22.122263 28916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:22.122283 28916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:22.122300 28916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:22.122315 28916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:22.127704 28916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:38413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:38099,127.6.31.189:41981,127.6.31.188:45555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.190.keytab
--principal=kudu/127.6.31.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.190:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.31.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:22.128741 28916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:22.129680 28916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:22.138121 28921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:22.138127 28922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:22.138284 28924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:22.138635 28916 server_base.cc:1034] running on GCE node
I20260927 10:54:22.139422 28916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:22.140579 28916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:22.141738 28916 hybrid_clock.cc:648] HybridClock initialized: now 1790506462141712 us; error 43 us; skew 500 ppm
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:22.144536 28916 init.cc:377] Logged in from keytab as kudu/127.6.31.190@KRBTEST.COM (short username kudu)
I20260927 10:54:22.145426 28916 webserver.cc:460] Webserver started at http://127.6.31.190:40757/ using document root <none> and password file <none>
I20260927 10:54:22.145963 28916 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:22.146028 28916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:22.146270 28916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:22.148312 28916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/data/instance:
uuid: "1e0867001a4a45c7990c421735f11ac4"
format_stamp: "Formatted at 2026-09-27 10:54:22 on dist-test-slave-c60x"
server_key: "482821d7cf13d85ab3fe1e8ca4ed02c0"
server_key_iv: "52d58233887621dd601b5e491c5ba50b"
server_key_version: "encryptionkey@0"
I20260927 10:54:22.148857 28916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/wal/instance:
uuid: "1e0867001a4a45c7990c421735f11ac4"
format_stamp: "Formatted at 2026-09-27 10:54:22 on dist-test-slave-c60x"
server_key: "482821d7cf13d85ab3fe1e8ca4ed02c0"
server_key_iv: "52d58233887621dd601b5e491c5ba50b"
server_key_version: "encryptionkey@0"
I20260927 10:54:22.152439 28916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:54:22.154776 28931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:22.156081 28916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:22.156247 28916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/wal
uuid: "1e0867001a4a45c7990c421735f11ac4"
format_stamp: "Formatted at 2026-09-27 10:54:22 on dist-test-slave-c60x"
server_key: "482821d7cf13d85ab3fe1e8ca4ed02c0"
server_key_iv: "52d58233887621dd601b5e491c5ba50b"
server_key_version: "encryptionkey@0"
I20260927 10:54:22.156361 28916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:22.170500 28916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:22.173631 28916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:22.173839 28916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:22.181424 28916 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.190:38099
I20260927 10:54:22.181747 28983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.190:38099 every 8 connection(s)
I20260927 10:54:22.182495 28916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:22.185350 28984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:22.189276 28984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:22.192517 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 28916
I20260927 10:54:22.192664 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-0/wal/instance
I20260927 10:54:22.192988 6270 external_mini_cluster.cc:1428] Setting key 62020bfde539f27099d434a68ec728ea
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.31.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.189@KRBTEST.COM" created.
I20260927 10:54:22.206103 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.192510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.6.31.190:38099)
0927 10:54:22.192903 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.192912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.192947 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:54:22.194036 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.194041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.194269 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.194421 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.195389 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.196824 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.197947 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.198650 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.198858 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.201314 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.201321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.201334 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.201358 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.202956 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.203468 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.203472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.203474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.203573 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.203957 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.203959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.203960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.204418 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.204556 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.205134 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.205334 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":283,"thread_start_us":95,"threads_started":1}
I20260927 10:54:22.206111 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.191750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38099 (local address 127.0.0.1:47854)
0927 10:54:22.192439 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.192502 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:22.193717 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:22.194473 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:22.194482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:22.194918 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:22.195239 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.195254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.196957 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.196959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.197824 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.197833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.198828 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.198831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.198916 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.199708 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:22.199724 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:22.201150 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:22.203059 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.203066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.203075 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.203367 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.203711 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.203713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.203715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.203860 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.204672 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:22.204678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:22.204851 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 10:54:22.205039 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 10:54:22.205439 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":118,"thread_start_us":134,"threads_started":1}
I20260927 10:54:22.208925 28984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:22.209762 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.209500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:41981 (local address 127.0.0.1:58242)
0927 10:54:22.209623 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.209703 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:54:22.209970 28932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
I20260927 10:54:22.210963 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.210734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:41981 (local address 127.0.0.1:58246)
0927 10:54:22.210859 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.210910 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 10:54:22.211161 28984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.189:41981: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab.
Entry for principal kudu/127.6.31.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab.
I20260927 10:54:22.223179 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.189:41981
--webserver_interface=127.6.31.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab
--principal=kudu/127.6.31.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:38413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:38099,127.6.31.189:41981,127.6.31.188:45555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:54:22.260414 28984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } attempt: 1
I20260927 10:54:22.261376 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.261040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:41981 (local address 127.0.0.1:58258)
0927 10:54:22.261215 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.261286 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:54:22.262084 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.261868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:41981 (local address 127.0.0.1:58270)
0927 10:54:22.261972 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.262022 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:54:22.262213 28984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.189:41981: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
I20260927 10:54:22.330389 28984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } attempt: 2
I20260927 10:54:22.331271 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.330919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:41981 (local address 127.0.0.1:58276)
0927 10:54:22.331111 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.331185 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 10:54:22.332008 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.331801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:41981 (local address 127.0.0.1:58280)
0927 10:54:22.331902 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.331948 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:54:22.332206 28984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.189:41981: Network error: Client connection negotiation failed: client connection to 127.6.31.189:41981: connect: Connection refused (error 111)
W20260927 10:54:22.333043 28991 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:54:22.333343 28991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:22.333385 28991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:22.333444 28991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:22.337649 28991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:22.337708 28991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:22.337728 28991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:22.337770 28991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:22.337805 28991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:22.343029 28991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:38413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:38099,127.6.31.189:41981,127.6.31.188:45555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.189.keytab
--principal=kudu/127.6.31.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.189:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.31.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:22.344128 28991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:22.345125 28991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:22.353154 28996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:22.353180 28999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:22.353135 28997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:22.353538 28991 server_base.cc:1034] running on GCE node
I20260927 10:54:22.354081 28991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:22.355154 28991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:22.356324 28991 hybrid_clock.cc:648] HybridClock initialized: now 1790506462356311 us; error 24 us; skew 500 ppm
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:22.358999 28991 init.cc:377] Logged in from keytab as kudu/127.6.31.189@KRBTEST.COM (short username kudu)
I20260927 10:54:22.359879 28991 webserver.cc:460] Webserver started at http://127.6.31.189:41211/ using document root <none> and password file <none>
I20260927 10:54:22.360446 28991 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:22.360539 28991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:22.360769 28991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:22.362810 28991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/data/instance:
uuid: "beb13cfbcca34f03b656fb5ea99f8a01"
format_stamp: "Formatted at 2026-09-27 10:54:22 on dist-test-slave-c60x"
server_key: "12fde87340685cae2a6a6dc815bdf591"
server_key_iv: "ad7cce97eafbb88a583a7cb3f935dc14"
server_key_version: "encryptionkey@0"
I20260927 10:54:22.363544 28991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/wal/instance:
uuid: "beb13cfbcca34f03b656fb5ea99f8a01"
format_stamp: "Formatted at 2026-09-27 10:54:22 on dist-test-slave-c60x"
server_key: "12fde87340685cae2a6a6dc815bdf591"
server_key_iv: "ad7cce97eafbb88a583a7cb3f935dc14"
server_key_version: "encryptionkey@0"
I20260927 10:54:22.367046 28991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:22.369500 29006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:22.370751 28991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:22.370884 28991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/wal
uuid: "beb13cfbcca34f03b656fb5ea99f8a01"
format_stamp: "Formatted at 2026-09-27 10:54:22 on dist-test-slave-c60x"
server_key: "12fde87340685cae2a6a6dc815bdf591"
server_key_iv: "ad7cce97eafbb88a583a7cb3f935dc14"
server_key_version: "encryptionkey@0"
I20260927 10:54:22.370992 28991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:22.392899 28991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:22.396242 28991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:22.396462 28991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:22.404228 28991 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.189:41981
I20260927 10:54:22.404572 29058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.189:41981 every 8 connection(s)
I20260927 10:54:22.405280 28991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/data/info.pb
I20260927 10:54:22.408237 29059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:22.409395 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 28991
I20260927 10:54:22.409567 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-1/wal/instance
I20260927 10:54:22.409860 6270 external_mini_cluster.cc:1428] Setting key 38d7c2596a427684004047e23f97dfbb
I20260927 10:54:22.412626 29059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.31.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.31.188@KRBTEST.COM" created.
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:54:22.423403 28984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } attempt: 3
I20260927 10:54:22.427448 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.414607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.6.31.190:38099)
0927 10:54:22.414766 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.414770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.415867 (+ 1097us) server_negotiation.cc:407] Connection header received
0927 10:54:22.416811 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.416815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.416877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.416994 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.418138 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.418679 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.420007 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.420279 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.420434 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.423044 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.423048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.423050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.423120 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.424892 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.425542 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.425547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.425549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.425633 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.426112 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.426116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.426117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.426326 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.426425 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.427162 (+ 737us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.427283 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:54:22.428645 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.415140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38099 (local address 127.0.0.1:47858)
0927 10:54:22.415704 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.415745 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:22.416605 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:22.417160 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:22.417166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:22.417501 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:22.417953 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.417965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.418818 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.418822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.419867 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.419877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.420474 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.420478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.420658 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.421323 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:22.421340 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:22.422852 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:22.425015 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.425024 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.425042 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.425425 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.425741 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.425746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.425751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.425956 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.426447 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:22.426455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:22.426833 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 10:54:22.427109 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 10:54:22.427506 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":157,"threads_started":1}
I20260927 10:54:22.430485 29059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
I20260927 10:54:22.435216 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.423881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:41981 (local address 127.0.0.1:58290)
0927 10:54:22.424101 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.424113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:22.424211 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:22.425193 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:22.425196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:22.425336 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:22.425439 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.425444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.426863 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.426868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.427677 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.427687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.428324 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.428327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.428435 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.429038 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:22.429049 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:22.431160 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:22.433308 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.433311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.433312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.433493 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.433808 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.433811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.433813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.433904 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.434652 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:22.434657 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:22.434770 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 10:54:22.434866 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:54:22.435000 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
I20260927 10:54:22.436291 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.424340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.6.31.189:41981)
0927 10:54:22.424687 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.424702 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.424729 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:54:22.424779 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.424786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.425024 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.425607 (+ 583us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.425738 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.426753 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.427811 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.428207 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.428325 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.431308 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.431316 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.431330 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.431355 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.433198 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.433595 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.433601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.433603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.433692 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.434016 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.434019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.434020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.434416 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.435554 (+ 1138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.435670 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.435837 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"spinlock_wait_cycles":6528,"thread_start_us":110,"threads_started":1}
I20260927 10:54:22.438112 28984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:22.439013 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.438787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54138)
0927 10:54:22.438889 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.438941 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.31.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab.
Entry for principal kudu/127.6.31.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab.
I20260927 10:54:22.439891 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.439589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54152)
0927 10:54:22.439766 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.439824 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 10:54:22.440044 28984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:45555: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
I20260927 10:54:22.440316 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.430973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:41981 (local address 127.0.0.1:58292)
0927 10:54:22.431105 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.431120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:22.431196 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:22.431653 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:22.431656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:22.431790 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:22.431929 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.431938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.432868 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.432871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.433635 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.433646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.434214 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.434218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.434322 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.434761 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:22.434775 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:22.436382 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:22.438187 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.438190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.438195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.438432 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.438901 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.438906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.438908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.438993 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.439765 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:22.439776 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:22.439877 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 10:54:22.439983 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:54:22.440133 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 10:54:22.440542 6270 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
/tmp/dist-test-taskh8OKcT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.188:45555
--webserver_interface=127.6.31.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab
--principal=kudu/127.6.31.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.31.148:38413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.31.190:38099,127.6.31.189:41981,127.6.31.188:45555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:54:22.440855 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.431074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.6.31.189:41981)
0927 10:54:22.431437 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.431442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.431451 (+ 9us) server_negotiation.cc:407] Connection header received
0927 10:54:22.431474 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.431506 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.431557 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.431649 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.432048 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.432719 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.433753 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.434095 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.434171 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.436541 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.436544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.436546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.436568 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.438092 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.438661 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.438664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.438665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.438795 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.439338 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.439341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.439343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.439588 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.439681 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.440514 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.440629 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":165,"threads_started":1}
I20260927 10:54:22.441481 29059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:22.442133 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.441900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54154)
0927 10:54:22.442003 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.442075 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:54:22.442430 29008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
I20260927 10:54:22.443387 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.443199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54170)
0927 10:54:22.443301 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.443341 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:54:22.443591 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:45555: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
I20260927 10:54:22.471205 28984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } attempt: 1
I20260927 10:54:22.472047 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.471764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54184)
0927 10:54:22.471907 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.471974 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 10:54:22.472874 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.472662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54186)
0927 10:54:22.472766 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.472812 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:54:22.473039 28984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:45555: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
I20260927 10:54:22.492846 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } attempt: 1
I20260927 10:54:22.493696 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.493411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54194)
0927 10:54:22.493553 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.493616 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 10:54:22.494393 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.494212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54206)
0927 10:54:22.494306 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.494346 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:54:22.494577 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:45555: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
I20260927 10:54:22.548234 28984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } attempt: 2
I20260927 10:54:22.549083 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.548783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54216)
0927 10:54:22.548944 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.549007 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 10:54:22.549875 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.549647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54224)
0927 10:54:22.549759 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.549811 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:54:22.550019 28984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:45555: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
W20260927 10:54:22.553378 29066 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:54:22.553632 29066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:22.553700 29066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:22.553761 29066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:22.557858 29066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:22.557910 29066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:22.557929 29066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:22.557945 29066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:22.558003 29066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:22.562808 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } attempt: 2
I20260927 10:54:22.563537 29066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.31.148:38413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.31.190:38099,127.6.31.189:41981,127.6.31.188:45555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/krb5kdc/kudu_127.6.31.188.keytab
--principal=kudu/127.6.31.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.31.188:45555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.31.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:22.563596 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.563328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54236)
0927 10:54:22.563470 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.563525 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 10:54:22.564397 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.564216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54250)
0927 10:54:22.564311 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.564351 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:54:22.564585 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.31.188:45555: Network error: Client connection negotiation failed: client connection to 127.6.31.188:45555: connect: Connection refused (error 111)
I20260927 10:54:22.564833 29066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:22.565805 29066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:22.574020 29073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:22.574008 29075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:22.574069 29072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:22.574286 29066 server_base.cc:1034] running on GCE node
I20260927 10:54:22.574914 29066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:22.576067 29066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:22.577232 29066 hybrid_clock.cc:648] HybridClock initialized: now 1790506462577214 us; error 29 us; skew 500 ppm
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:22.580001 29066 init.cc:377] Logged in from keytab as kudu/127.6.31.188@KRBTEST.COM (short username kudu)
I20260927 10:54:22.580906 29066 webserver.cc:460] Webserver started at http://127.6.31.188:43127/ using document root <none> and password file <none>
I20260927 10:54:22.581482 29066 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:22.581573 29066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:22.581809 29066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:22.583874 29066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/data/instance:
uuid: "b7c47a838cb34628bbc9de3f94597b14"
format_stamp: "Formatted at 2026-09-27 10:54:22 on dist-test-slave-c60x"
server_key: "dc9b57fea4d24451a805323a27d3a533"
server_key_iv: "41c90c8be8d7696171c702999da0fa02"
server_key_version: "encryptionkey@0"
I20260927 10:54:22.584421 29066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/wal/instance:
uuid: "b7c47a838cb34628bbc9de3f94597b14"
format_stamp: "Formatted at 2026-09-27 10:54:22 on dist-test-slave-c60x"
server_key: "dc9b57fea4d24451a805323a27d3a533"
server_key_iv: "41c90c8be8d7696171c702999da0fa02"
server_key_version: "encryptionkey@0"
I20260927 10:54:22.588016 29066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:54:22.590277 29082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:22.591533 29066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:22.591668 29066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/wal
uuid: "b7c47a838cb34628bbc9de3f94597b14"
format_stamp: "Formatted at 2026-09-27 10:54:22 on dist-test-slave-c60x"
server_key: "dc9b57fea4d24451a805323a27d3a533"
server_key_iv: "41c90c8be8d7696171c702999da0fa02"
server_key_version: "encryptionkey@0"
I20260927 10:54:22.591773 29066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:22.603821 29066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:22.607247 29066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:22.607447 29066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:22.614338 29066 rpc_server.cc:307] RPC server started. Bound to: 127.6.31.188:45555
I20260927 10:54:22.614722 29134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.31.188:45555 every 8 connection(s)
I20260927 10:54:22.615329 29066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/data/info.pb
I20260927 10:54:22.617357 6270 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu as pid 29066
I20260927 10:54:22.617513 6270 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8OKcT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506175715859-6270-0/minicluster-data/master-2/wal/instance
I20260927 10:54:22.617825 6270 external_mini_cluster.cc:1428] Setting key f6b17dd48ef86e7b822f18100df98f19
I20260927 10:54:22.618322 29135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:22.623411 29135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:54:22.634276 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.619265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.6.31.188:45555)
0927 10:54:22.619676 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.619700 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.619743 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:54:22.620573 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.620595 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.621031 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.621405 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.622093 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.623608 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.624615 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.625247 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.625412 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.628336 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.628342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.628355 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.628386 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.630692 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.631266 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.631275 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.631282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.631422 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.631764 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.631772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.631773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.632185 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.632404 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.632729 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.633137 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":104,"threads_started":1}
I20260927 10:54:22.635306 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.625329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.190:38099 (local address 127.0.0.1:47874)
0927 10:54:22.625814 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.625870 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:22.626011 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:22.626340 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:22.626345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:22.626836 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:22.627005 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.627014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.627808 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.627813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.628697 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.628715 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.629257 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.629264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.629413 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.629852 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:22.629872 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:22.631630 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:22.633431 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.633438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.633443 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.633708 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.633983 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.633987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.633988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.634080 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.634666 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:22.634673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:22.634785 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 10:54:22.634924 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 10:54:22.635112 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":116,"threads_started":1}
I20260927 10:54:22.635540 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.625429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.6.31.190:38099)
0927 10:54:22.625573 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.625577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.625957 (+ 380us) server_negotiation.cc:407] Connection header received
0927 10:54:22.626136 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.626139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.626183 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.626250 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.627193 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.627669 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.628820 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.629123 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.629217 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.631790 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.631794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.631796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.631821 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.633323 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.633811 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.633813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.633815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.633891 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.634219 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.634221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.634223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.634516 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.634616 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.635228 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.635342 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:54:22.636524 29135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.190@KRBTEST.COM
I20260927 10:54:22.645174 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.637224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.6.31.189:41981)
0927 10:54:22.637349 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.637353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.637367 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:54:22.637416 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.637419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.637468 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.637578 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.637962 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.638597 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.639450 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.639767 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.639846 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.641878 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.641881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.641883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.641904 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.643540 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.643996 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.644001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.644003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.644086 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.644357 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.644362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.644364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.644534 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.644656 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.644914 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.645028 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:54:22.645196 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.637025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.189:41981 (local address 127.0.0.1:58302)
0927 10:54:22.637140 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.637156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:22.637269 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:22.637599 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:22.637602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:22.637760 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:22.637846 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.637851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.638704 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.638707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.639342 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.639350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.639875 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.639881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.639978 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.640465 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:22.640481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:22.641726 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:22.643686 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.643689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.643698 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.643893 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.644179 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.644184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.644186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.644267 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.644646 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:22.644649 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:22.644747 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 10:54:22.644853 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:54:22.644981 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 10:54:22.645946 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.637107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.6.31.190:38099)
0927 10:54:22.637208 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.637214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.637332 (+ 118us) server_negotiation.cc:407] Connection header received
0927 10:54:22.637444 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.637446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.637481 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.637563 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.638241 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.638745 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.639588 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.639792 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.639944 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.642597 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.642603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.642610 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.642631 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.644103 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.644534 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.644539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.644541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.644621 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.644894 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.644899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.644901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.645066 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.645701 (+ 635us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.645763 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.645845 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:54:22.646122 29135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:54:22.649197 28984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } attempt: 3
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.188@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.31.189@KRBTEST.COM
I20260927 10:54:22.654369 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.646643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.6.31.188:45555)
0927 10:54:22.646754 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.646758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.646771 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:54:22.646893 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.646895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.646953 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.647026 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.647454 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.647945 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.648634 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.648877 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.648980 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.650912 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.650916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.650918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.650940 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.652717 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.653116 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.653122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.653124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.653241 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.653511 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.653517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.653519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.653718 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.653804 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.654077 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.654201 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":51}
I20260927 10:54:22.654588 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.646605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54268)
0927 10:54:22.646699 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.646713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:22.646781 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:22.647167 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:22.647170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:22.647291 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:22.647350 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.647354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.648043 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.648049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.648529 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.648535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.648977 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.648980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.649074 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.649531 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:22.649549 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:22.650782 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:22.652834 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.652837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.652839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.653019 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.653347 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.653350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.653352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.653428 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.653836 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:22.653838 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:22.653910 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 10:54:22.654005 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:54:22.654125 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":4}
I20260927 10:54:22.655752 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } attempt: 3
I20260927 10:54:22.657630 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.647073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.6.31.189:41981)
0927 10:54:22.647230 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.647237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.647250 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:54:22.647358 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.647361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.647397 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.647489 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.648152 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.648620 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.649540 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.649809 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.650234 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.653696 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.653699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.653702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.653727 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.655954 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.656394 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.656399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.656400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.656494 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.656772 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.656776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.656778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.656952 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.657176 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.657308 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.657418 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 10:54:22.658411 29135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14: Bootstrap starting.
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.190@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
I20260927 10:54:22.662729 29135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:22.663451 6270 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:22.663674 29135 log.cc:824] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:22.664335 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.649688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54276)
0927 10:54:22.649816 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.649830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:22.649901 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:22.650604 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:22.650608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:22.651059 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:22.651207 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.651218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.653607 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.653611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.654391 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.654401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.655046 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.655050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.655154 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.655617 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:22.655633 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:22.660656 (+ 5023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:22.662637 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.662643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.662645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.662895 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.663238 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.663241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.663244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.663331 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.663778 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:22.663782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:22.663885 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 10:54:22.663998 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:54:22.664147 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
Sep 27 10:54:22 dist-test-slave-c60x krb5kdc[28903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.31.189@KRBTEST.COM for kudu/127.6.31.188@KRBTEST.COM
I20260927 10:54:22.665004 29140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.649961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.6.31.188:45555)
0927 10:54:22.650307 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.650317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.650329 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:54:22.650357 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.650360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.650397 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.650586 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.651660 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.653500 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.654532 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.654918 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.655375 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.660802 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.660806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.660808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.660831 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.662514 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.663036 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.663039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.663040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.663155 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.663435 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.663439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.663442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.663659 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.664596 (+ 937us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.664673 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.664787 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"spinlock_wait_cycles":2944,"thread_start_us":137,"threads_started":1}
I20260927 10:54:22.665129 28958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1e0867001a4a45c7990c421735f11ac4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47880
I20260927 10:54:22.668063 28984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4: Bootstrap starting.
I20260927 10:54:22.668174 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.656163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.31.188:45555 (local address 127.0.0.1:54284)
0927 10:54:22.659403 (+ 3240us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:22.659414 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:22.659520 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:22.659778 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:22.659781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:22.659972 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:22.660073 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.660084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.661156 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.661162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.662054 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:22.662068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.662716 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.662723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:22.662814 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.663488 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:22.663503 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:22.664854 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:22.666639 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.666642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.666644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.666882 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.667206 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:22.667209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:22.667211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:22.667296 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:22.667724 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:22.667728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:22.667818 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 10:54:22.667898 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:54:22.668030 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3181}
I20260927 10:54:22.668226 29135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14: No bootstrap required, opened a new log
I20260927 10:54:22.668767 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.656190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.6.31.188:45555)
0927 10:54:22.656294 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.656298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.659493 (+ 3195us) server_negotiation.cc:407] Connection header received
0927 10:54:22.659606 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.659609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.659652 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.660229 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.660319 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.661024 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.662209 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.662596 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.663025 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.664993 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.665000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.665004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.665027 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.666518 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.666994 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.666997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.666999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.667076 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.667402 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.667405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.667406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.667609 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.668431 (+ 822us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.668521 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.668631 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":1408}
I20260927 10:54:22.670753 28984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:22.671542 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01: Bootstrap starting.
I20260927 10:54:22.671738 28984 log.cc:824] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:22.672273 29135 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } }
I20260927 10:54:22.672511 29135 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c47a838cb34628bbc9de3f94597b14, State: Initialized, Role: FOLLOWER
I20260927 10:54:22.672863 29135 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } }
I20260927 10:54:22.673489 29143 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.673802 29143 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:22.674044 29059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:22.674237 29135 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:22.674832 28984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4: No bootstrap required, opened a new log
I20260927 10:54:22.674870 29059 log.cc:824] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:22.677302 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01: No bootstrap required, opened a new log
I20260927 10:54:22.678107 28984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } }
I20260927 10:54:22.678331 28984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0867001a4a45c7990c421735f11ac4, State: Initialized, Role: FOLLOWER
I20260927 10:54:22.678699 28984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } }
I20260927 10:54:22.679191 29156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.679360 29156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:22.679947 28984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:22.680025 28958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:54:22.680152 28958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:22.681422 28958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } }
I20260927 10:54:22.678442 29066 master_runner.cc:186] Error getting master registration for 127.6.31.190:38099: OK, instance_id { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" instance_seqno: 1790506462179394 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:54:22.682490 29066 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:54:22.683491 29109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e0867001a4a45c7990c421735f11ac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7c47a838cb34628bbc9de3f94597b14"
I20260927 10:54:22.683707 29109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [term 0 FOLLOWER]: Advancing to term 1
W20260927 10:54:22.684463 29155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:22.684536 29155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:22.685015 29109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0867001a4a45c7990c421735f11ac4 in term 1.
W20260927 10:54:22.685324 29167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:22.685451 29167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:22.682369 28958 leader_election.cc:288] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [CANDIDATE]: Term 1 election: Requested vote from peers beb13cfbcca34f03b656fb5ea99f8a01 (127.6.31.189:41981), b7c47a838cb34628bbc9de3f94597b14 (127.6.31.188:45555)
I20260927 10:54:22.685901 28933 leader_election.cc:302] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e0867001a4a45c7990c421735f11ac4, b7c47a838cb34628bbc9de3f94597b14; no voters:
I20260927 10:54:22.686153 29156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:22.686367 29156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [term 1 LEADER]: Becoming Leader. State: Replica: 1e0867001a4a45c7990c421735f11ac4, State: Running, Role: LEADER
I20260927 10:54:22.686664 29156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } }
I20260927 10:54:22.688388 29156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e0867001a4a45c7990c421735f11ac4. Latest consensus state: current_term: 1 leader_uuid: "1e0867001a4a45c7990c421735f11ac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.688504 29156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:22.686077 28916 master_runner.cc:186] Error getting master registration for 127.6.31.189:41981: OK, instance_id { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" instance_seqno: 1790506462402023 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:54:22.689317 28916 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:54:22.690588 29176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:22.690822 29059 raft_consensus.cc:348] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } }
I20260927 10:54:22.691234 29059 raft_consensus.cc:729] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beb13cfbcca34f03b656fb5ea99f8a01, State: Initialized, Role: FOLLOWER
I20260927 10:54:22.691581 29176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:22.691747 29059 consensus_queue.cc:260] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } }
I20260927 10:54:22.693845 29059 sys_catalog.cc:564] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:22.693886 29033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e0867001a4a45c7990c421735f11ac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "beb13cfbcca34f03b656fb5ea99f8a01"
I20260927 10:54:22.694130 29033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:22.695605 29033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0867001a4a45c7990c421735f11ac4 in term 1.
W20260927 10:54:22.699467 29189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:22.699579 29189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:22.703533 29177 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.703666 29177 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:22.707029 29109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [term 1 FOLLOWER]: Refusing update from remote peer 1e0867001a4a45c7990c421735f11ac4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:22.707770 29156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:22.711908 29143 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e0867001a4a45c7990c421735f11ac4. Latest consensus state: current_term: 1 leader_uuid: "1e0867001a4a45c7990c421735f11ac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.712049 29143 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:22.713052 29033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [term 1 FOLLOWER]: Refusing update from remote peer 1e0867001a4a45c7990c421735f11ac4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:22.713629 29156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:22.717376 29177 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e0867001a4a45c7990c421735f11ac4. Latest consensus state: current_term: 1 leader_uuid: "1e0867001a4a45c7990c421735f11ac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.717504 29177 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:22.721503 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.699018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.6.31.189:41981)
0927 10:54:22.699135 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.699139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.700902 (+ 1763us) server_negotiation.cc:407] Connection header received
0927 10:54:22.701071 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.701074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.701117 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.701167 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.702100 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.702800 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.707506 (+ 4706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.707890 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.707969 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.712366 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.712370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.712373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.712405 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.714594 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.718097 (+ 3503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.718101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.718103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.718187 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.719316 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.719320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.719322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.719542 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.719663 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.721216 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.721346 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:54:22.723284 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.698374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.6.31.188:45555)
0927 10:54:22.698502 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.698506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.699994 (+ 1488us) server_negotiation.cc:407] Connection header received
0927 10:54:22.700170 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.700177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.700228 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.700280 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.703681 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.704450 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.708452 (+ 4002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.708816 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.708890 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.715880 (+ 6990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.715884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.715887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.715917 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.717644 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.718474 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.718477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.718479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.718558 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.719949 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.719952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.719954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.720164 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.720271 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.722992 (+ 2721us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.723133 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:54:22.723282 29170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e0867001a4a45c7990c421735f11ac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.723423 29170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:22.724668 29177 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e0867001a4a45c7990c421735f11ac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.724783 29177 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:22.724921 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:22.697672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.6.31.190:38099)
0927 10:54:22.697782 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.697786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.699631 (+ 1845us) server_negotiation.cc:407] Connection header received
0927 10:54:22.699845 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.699849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.699915 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.700006 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.701612 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.702393 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.706463 (+ 4070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:22.706826 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:22.706908 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:22.712790 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:22.712797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:22.712802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:22.712833 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:22.715381 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.718925 (+ 3544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.718931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.718934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.719015 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:22.720543 (+ 1528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:22.720547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:22.720549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:22.720777 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:22.720893 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:22.724648 (+ 3755us) server_negotiation.cc:299] Negotiation successful
0927 10:54:22.724771 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:54:22.725165 29178 mvcc.cc:205] Tried to move back new op lower bound from 7333914471239581696 to 7333914471167901696. Current Snapshot: MvccSnapshot[applied={T|T < 7333914471239581696}]
I20260927 10:54:22.727898 29156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e0867001a4a45c7990c421735f11ac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.728025 29156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:22.727562 29196 mvcc.cc:205] Tried to move back new op lower bound from 7333914471239581696 to 7333914471167901696. Current Snapshot: MvccSnapshot[applied={T|T < 7333914471239581696}]
I20260927 10:54:22.730998 29143 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e0867001a4a45c7990c421735f11ac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0867001a4a45c7990c421735f11ac4" member_type: VOTER last_known_addr { host: "127.6.31.190" port: 38099 } } peers { permanent_uuid: "beb13cfbcca34f03b656fb5ea99f8a01" member_type: VOTER last_known_addr { host: "127.6.31.189" port: 41981 } } peers { permanent_uuid: "b7c47a838cb34628bbc9de3f94597b14" member_type: VOTER last_known_addr { host: "127.6.31.188" port: 45555 } } }
I20260927 10:54:22.731142 29143 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:54:22.731551 29194 mvcc.cc:205] Tried to move back new op lower bound from 7333914471239581696 to 7333914471167901696. Current Snapshot: MvccSnapshot[applied={T|T < 7333914471239581696}]
I20260927 10:54:22.732579 29176 catalog_manager.cc:1347] Generated new cluster ID: c330efc84c0d44619b0243e58f58101a
I20260927 10:54:22.732652 29176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:22.766014 29176 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:22.767387 29176 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:22.786041 29176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e0867001a4a45c7990c421735f11ac4: Generated new TSK 0
I20260927 10:54:22.787134 29176 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:54:22.821869 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.818652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.6.31.190:38099)
0927 10:54:22.818782 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.818787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.820686 (+ 1899us) server_negotiation.cc:407] Connection header received
0927 10:54:22.820894 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.820898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.820951 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.821004 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.821726 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47890: BlockingRecv error: recv got EOF from 127.0.0.1:47890 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 10:54:22.823822 29138 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.819404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.6.31.188:45555)
0927 10:54:22.819517 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.819521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.822520 (+ 2999us) server_negotiation.cc:407] Connection header received
0927 10:54:22.822709 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.822712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.822763 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.822815 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.823696 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54310: BlockingRecv error: recv got EOF from 127.0.0.1:54310 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 10:54:22.825002 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.820094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.6.31.189:41981)
0927 10:54:22.820211 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.820215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.823312 (+ 3097us) server_negotiation.cc:407] Connection header received
0927 10:54:22.824237 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.824241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.824296 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.824349 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.824891 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58340: BlockingRecv error: recv got EOF from 127.0.0.1:58340 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:54:22.829749 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.828294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.6.31.189:41981)
0927 10:54:22.828396 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.828400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.828808 (+ 408us) server_negotiation.cc:407] Connection header received
0927 10:54:22.829003 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.829008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.829093 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.829146 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.829654 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58348: BlockingRecv error: recv got EOF from 127.0.0.1:58348 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 10:54:22.834748 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.833347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.6.31.189:41981)
0927 10:54:22.833438 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.833442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.833886 (+ 444us) server_negotiation.cc:407] Connection header received
0927 10:54:22.834060 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.834064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.834113 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.834164 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.834651 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58364: BlockingRecv error: recv got EOF from 127.0.0.1:58364 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 10:54:22.840930 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.839450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.6.31.189:41981)
0927 10:54:22.839544 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.839548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.839987 (+ 439us) server_negotiation.cc:407] Connection header received
0927 10:54:22.840186 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.840190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.840243 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.840297 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.840831 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58378: BlockingRecv error: recv got EOF from 127.0.0.1:58378 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 10:54:22.849104 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.847445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.6.31.189:41981)
0927 10:54:22.847643 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.847647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.848144 (+ 497us) server_negotiation.cc:407] Connection header received
0927 10:54:22.848334 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.848337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.848388 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.848441 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.848961 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58394: BlockingRecv error: recv got EOF from 127.0.0.1:58394 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260927 10:54:22.862066 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.860169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.6.31.189:41981)
0927 10:54:22.860300 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.860305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.860957 (+ 652us) server_negotiation.cc:407] Connection header received
0927 10:54:22.861222 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.861226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.861286 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.861346 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.861941 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58402: BlockingRecv error: recv got EOF from 127.0.0.1:58402 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 10:54:22.882833 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.881052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.6.31.189:41981)
0927 10:54:22.881191 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.881196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.881750 (+ 554us) server_negotiation.cc:407] Connection header received
0927 10:54:22.881962 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.881966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.882027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.882086 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.882709 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58404: BlockingRecv error: recv got EOF from 127.0.0.1:58404 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 10:54:22.918756 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.917677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.6.31.189:41981)
0927 10:54:22.917824 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.917828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.917883 (+ 55us) server_negotiation.cc:407] Connection header received
0927 10:54:22.918008 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.918013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.918073 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.918199 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.918634 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58410: BlockingRecv error: recv got EOF from 127.0.0.1:58410 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 10:54:22.985973 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:22.984851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.6.31.189:41981)
0927 10:54:22.985027 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:22.985032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:22.985094 (+ 62us) server_negotiation.cc:407] Connection header received
0927 10:54:22.985221 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:22.985224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:22.985273 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:22.985350 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:22.985821 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58416: BlockingRecv error: recv got EOF from 127.0.0.1:58416 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 10:54:23.117175 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:23.116079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.6.31.189:41981)
0927 10:54:23.116248 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:23.116252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:23.116268 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:54:23.116354 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:23.116356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:23.116401 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:23.116474 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:23.116966 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58428: BlockingRecv error: recv got EOF from 127.0.0.1:58428 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 10:54:23.377084 29065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:23.375660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.6.31.189:41981)
0927 10:54:23.375805 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:23.375810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:23.376132 (+ 322us) server_negotiation.cc:407] Connection header received
0927 10:54:23.376289 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:23.376292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:23.376350 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:23.376433 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:23.376924 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58438: BlockingRecv error: recv got EOF from 127.0.0.1:58438 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:54:23.685807 29155 catalog_manager.cc:1259] Loaded cluster ID: c330efc84c0d44619b0243e58f58101a
I20260927 10:54:23.685890 29155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14: loading cluster ID for follower catalog manager: success
I20260927 10:54:23.687513 29155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14: acquiring CA information for follower catalog manager: success
I20260927 10:54:23.688618 29155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7c47a838cb34628bbc9de3f94597b14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:54:23.700783 29189 catalog_manager.cc:1259] Loaded cluster ID: c330efc84c0d44619b0243e58f58101a
I20260927 10:54:23.700882 29189 catalog_manager.cc:1552] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01: loading cluster ID for follower catalog manager: success
I20260927 10:54:23.702453 29189 catalog_manager.cc:1574] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01: acquiring CA information for follower catalog manager: success
I20260927 10:54:23.703677 29189 catalog_manager.cc:1602] T 00000000000000000000000000000000 P beb13cfbcca34f03b656fb5ea99f8a01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:54:23.895002 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:23.891192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.6.31.189:41981)
0927 10:54:23.891442 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:23.891445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:23.891460 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:54:23.891511 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:23.891513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:23.891641 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:23.891771 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:23.892087 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.892604 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.893435 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.893650 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.893737 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:23.894016 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:23.894572 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:23.894734 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:23.894806 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 10:54:23.894871 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260927 10:54:23.902118 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:23.898170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.6.31.188:45555)
0927 10:54:23.898396 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:23.898399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:23.898412 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:54:23.898515 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:23.898517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:23.898631 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:23.898727 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:23.898917 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.899653 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.900721 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.900946 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.901026 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:23.901286 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:23.901719 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:23.901807 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:23.901875 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 10:54:23.901919 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":59,"threads_started":1}
I20260927 10:54:23.909684 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:23.905676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.6.31.188:45555)
0927 10:54:23.905803 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:23.905813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:23.906086 (+ 273us) server_negotiation.cc:407] Connection header received
0927 10:54:23.906200 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:23.906203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:23.906248 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:23.906332 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:23.906563 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.907369 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.908534 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.908804 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.908908 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:23.909232 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:23.909299 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:23.909371 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:23.909481 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 10:54:23.909533 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:54:23.910055 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:23.905609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.6.31.189:41981)
0927 10:54:23.905756 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:23.905760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:23.905779 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:54:23.906057 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:23.906061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:23.906121 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:23.906192 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:23.906563 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.907192 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.908439 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.908784 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.908883 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:23.909181 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:23.909282 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:23.909354 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:23.909889 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 10:54:23.909947 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:54:23.910689 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:23.905341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.6.31.190:38099)
0927 10:54:23.905575 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:23.905580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:23.905954 (+ 374us) server_negotiation.cc:407] Connection header received
0927 10:54:23.906082 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:23.906085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:23.906257 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:23.906370 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:23.906710 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.907507 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.908341 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:23.908667 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:23.908765 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:23.909126 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:23.909522 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:23.909601 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:23.910220 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 10:54:23.910284 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260927 10:54:23.913926 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 28916
I20260927 10:54:23.920869 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 28991
I20260927 10:54:23.926638 6270 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8OKcT/build/debug/bin/kudu with pid 29066
2026-09-27T10:54:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3104 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6268 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288204 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST